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PREFACE 


This  guide  was  written  to  provide  practical  assistance  to  people  everywhere  who  need 
help  developing  surveys.  It  should  be  useful  to  people  inside  and  outside  the  Air  Force  who  are 
interested  in  developing  and  analyzing  a  survey.  It  is  also  useful  for  survey  participants  as  well 
as  survey  developers.  I  was  inspired  to  write  it  by  the  hundreds  of  people  who  have  come  to  me 
for  survey  advice  and  assistance  over  the  past  several  years. 

I  take  a  strong  stand  on  some  issues  and  readers  may  not  agree  with  me.  Also,  my  views 
do  not  necessarily  reflect  those  of  the  Armstrong  Laboratory,  the  Air  Force  or  the  United  States 
Government. 

I  thank  my  colleagues  in  the  Human  Resources  Directorate  of  the  Armstrong  Laboratory 
for  their  valuable  comments  on  this  manuscript.  Special  thanks  go  to  Dr.  Walter  G.  Albert,  Dr. 

R.  Bruce  Gould,  Col.  Gary  D.  Zank,  Mr.  Larry  T.  Looper,  Dr.  Patrick  C.  Kyllonen  and  Dr.  M. 
Jacobina  Skinner.  Dr.  Albert  was  especially  helpful.  He  frequently  provided  helpful  comments 
and  gave  the  manuscript  a  thorough  final  edit.  Mr.  William  J.  Phalen,  formerly  of  the 
Directorate,  also  provided  good  advice  on  scaling  issues.  Ms.  Virginia  Moody  of  the  Human 
Systems  Center  Quality  Office  encouraged  me  to  write  the  manuscript,  gave  it  a  careful  review, 
and  provided  excellent  suggestions.  Mr.  Charles  Hamilton  and  Mr.  Louis  Datko,  survey  experts 
at  Headquarters  Air  Force  Personnel  Center  (HQ  AFPC),  also  reviewed  the  manuscript  and 
provided  valuable  input. 

In  the  ethics  chapter  and  in  a  few  appendices  there  is  discussion  of  legal  and 
Constitutional  issues.  I  thank  Judge  Arthur  J.  Doran  Jr.  and  attorney  Arthur  J.  Doran  III  for 
reviewing  the  manuscript  and  attesting  to  the  veracity  of  the  legal  discourse.  I  thank  Ms.  Jane 
Tuck  for  inspiring  his  devotion  to  the  Constitution  and  to  the  Bill  of  Rights. 

I  thank  Ms.  Sharon  L.  McDonald  and  Ms.  Kathryn  J.  Hawk  of  the  Human  Resources 
Directorate  who  often  helped  me  with  word-processing  problems.  Sharon  also  edited  a  draft  of 
the  report  and  found  mistakes  I  would  not  have  found  himself.  Thanks  also  go  to  Ms.  Elizabeth 
M.  Knippa  for  reviewing  an  earlier  draft. 

Ms.  Marilyn  Goff,  Ms.  Deborah  A.  Hann  and  Ms.  Laura  Gomez  of  the  Technical  Library 
at  the  Human  Resources  Directorate  also  provided  valuable  assistance  by  ordering  and  returning 
numerous  books  through  interlibrary  loan  and  tracking  down  obscure  information  in  the  library. 

Use  of  First  Person  Pronouns;  Notice  that  I  use  “I”  and  “me”  above.  It  is  uncommon 
to  write  technical  reports  in  the  first  person  using  personal  pronouns.  However,  this  report  is  in 
large  part  a  personal  expression  of  the  author’s  experiences  in  research  and  consultation  over 
several  decades  in  which  he  provides  personal  examples.  Initially  “the  author”  was  used.  This 
term  was  awkward  and  sounded  stilted.  Thus,  throughout  the  report,  IdiMe  now  use  “I  ”  “me”  or 
“my”  to  refer  to  myself. 
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GUTOELINES  FOR  CONDUCTING  SURVEYS 


Chapter  1 
Introduction 

About  six  years  ago  I  was  asked  by  the  commander  of  what  was  then  the  Human 
Systems  Center  to  develop  customer  satisfaction  instruments  for  the  center.  My  colleagues  and  I 
were  also  asked  to  train  others  working  in  quality  improvement  offices  throughout  what  was  then 
Air  Force  Systems  Command  (AFSC)  on  how  to  build  such  surveys  themselves.  I  produced  a 
brief  handout  at  that  time  called  Guidelines  for  Building  Customer  Satisfaction  Instruments 
which  people  have  asked  for  ever  since.  This  handout  was  very  short,  providing  only  limited 
guidance.  Guidelines  for  Constructing  Surveys  was  written  to  provide  far  more  complete 
information. 

The  intent  is  to  provide  a  practical  teaching  tool  useful  to  anyone,  inside  or  outside  the 
Air  Force,  who  wants  to  develop,  administer  and  analyze  a  survey.  Although  it  is  intended  to 
help  Air  Force  personnel,  it  has  broad  applicability  to  survey  researchers  anywhere.  The  text  is 
most  useful  to  survey  developers  who  wish  to  measure  general  attitudes  or  behaviors.  It  is  less 
applicable  to  occupational  analysts  concerned  with  surveys  of  job  tasks. 

Hundreds  of  people  requested  my  help  with  surveys  over  the  past  several  years  and  I 
had  little  written  guidance  to  offer  them.  Good  academic  survey  texts,  such  as  the  Handbook  of 
Survey  Research  (Rossi,  Wright  &  Anderson,  1983),  were  usually  considered  too  technical  and 
difficult  to  be  of  much  use  to  the  people  who  asked  for  assistance.  They  wanted  rapid  access  to 
easily  understood,  practical  information 

Although  many  survey  texts  may  be  too  complex  for  would-be  survey  developers  who 
are  new  to  the  task,  there  are  some  easily  understood  books  available  which  may  also  be  helpful. 
See  for  example,  Fink  and  Kosecoff  (1985),  Bradbum  and  Sudman  (1988),  Fowler  (1993)  and 
Salant  and  Dillman  (1994). 

This  text  should  not  only  be  helpful  to  people  conducting  surveys,  but  also  to  those  who 
are  asked  to  take  a  survey.  It  will  help  both  survey  developers  and  potential  survey  participants 
tell  the  difference  between  well  constructed  and  poorly  constructed  surveys.  Also,  I  hope  to 
heighten  the  sensitivity  of  data  gatherers  and  information  providers  to  ethical  issues.  Survey 
developers  will  learn  about  their  ethical  responsibilities  to  survey  participants.  Participants  will 
learn  under  what  conditions  they  may  have  the  right  to  refuse  to  participate  if  survey  developers, 
out  of  ignorance  or  insensitivity,  do  not  live  up  to  these  responsibilities  (see  Chapter  4). 

Many  examples  of  items  or  other  survey  content  are  provided.  These  are  placed 
in  a  simple  border  like  this: 


Examples  of  items  and  other  survey  content  are  framed  like  this. 


Sometimes  advanced  topics  are  discussed  that  might  not  be  of  interest  to  readers  who 
need  only  develop  and  analyze  a  simple  survey.  Automated  surveys  and  inferential  analyses  fall 
into  this  category.  Such  optional  reading  is  placed  in  a  shadowbox  like  this: 
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Eleven  appendices  are  provided.  They  contain  information  that  may  distract  from  the 
flow  of  the  text  and  be  only  of  interest  to  some  readers.  Their  content  is  as  follows: 

•  Appendix  A  contains  examples  of  cover  letters  from  two  surveys. 

•  Appendix  B  contains  examples  of  introductory  comments  and  instructions. 

•  Appendix  C  contains  a  discussion  of  the  rights  perspective  which  provides  a  foundation  for 
the  survey  ethics  chapter. 

•  Appendix  D  provides  ethical  principles  which  are  relevant  to  surveys  from  the  American 
Psychological  Association’s  (APA’s)  Ethics  Code. 

•  Appendix  E  provides  a  First  Amendment  discussion  of  why  specific  information  about  job- 
irrelevant  organizational  memberships  should  be  avoided. 

•  Appendix  F  provides  guidance  for  group  survey  administrators. 

•  Appendix  G  provides  a  Statement  of  Work  requesting  automated  survey  disk  duplication  and 
distribution  under  contract. 

•  Appendix  H  provides  guidance  for  automated  survey  points  of  contact. 

•  Appendix  I  provides  examples  of  many  different  Likert  scales. 

•  Appendix  J  provides  brief  guidance  on  interviewing  which  would  be  useful,  for  example,  if 
you  were  interviewing  to  determine  survey  content  or  interviewing  pretest  participants. 

•  Appendix  K  provides  suggested  questions  for  pretest  participants. 

The  text  contains  intentional  redundancies.  This  is  because  readers  are  expected  to  enter 
and  exit  the  manuscript  at  any  point  that  provides  useful  content.  Also,  they  may  or  may  not 
read  what  is  in  the  shadowboxes.  The  contents  pages  are  quite  detailed  so  readers  can  easily  find 
what  they  want.  If  you  do  not  have  time  to  read  the  text,  but  are  interested  in  a  summary  of  the 
recommendations,  go  to  the  Postscript  after  Chapter  10.  The  Postscript  also  provides  a  good 
overview  of  the  content  of  each  chapter. 

In  some  situations,  a  survey  might  not  be  the  best  approach  to  gathering  information. 

For  instance,  you  may  need  in-depth  information  from  a  small  group  of  people  and  care  little 
about  quantitative  analyses.  In  such  a  situation,  an  interview  approach  would  be  more  suitable. 
You  may  want  to  use  multiple  methods.  For  example,  you  may  want  to  conduct  interviews  as  an 
initial  step  to  gather  information  concerning  the  appropriate  content  of  a  survey.  For  those 
interested  in  interviewing  techniques,  see  the  companion  manuscript.  Guidelines  for 
Conducting  Interviews  (Watson,  1997).  You  will  also  find  brief  guidance  on  interviewing  in 
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Appendix  J.  You  may  want  to  seek  the  assistance  of  organizations  which  conduct  surveys  or 
interviews.  Bradbum  and  Sudman  (1988,  Chapter  4)  provide  an  extensive  list. 

You  may  want  to  take  a  more  anthropological  approach.  You  could  place  yourself  more 
fully  in  a  setting  to  get  a  feel  for  its  “culture.”  For  those  interested  in  naturalistic  observation, 
see  Patton’s  (1980)  Chapter  6,  Evaluation  Through  Observation,  or  Jorgensen  (1989). 
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Chapter  2 


Paper-and-Pencil,  Automated,  and  Telephone  Surveys 

Surveys  come  in  several  forms,  from  paper-and-pencil  to  automated  or  computerized 
formats.  Sometimes  an  interviewer  conducts  a  survey  by  phone.  Each  of  these  approaches  is 
discussed  below. 

Paper-and-Pencil  Surveys 

Paper-and-pencil  surveys  have  been  used  for  years.  They  are  simple  to  develop,  but 
printing  can  be  time  consuming  and  administration,  especially  via  mail,  can  be  cumbersome. 
They  usually  require  a  separate  or  embedded  optical  scan  sheet,  which  needs  to  be  scanned 
before  a  data  tape  can  be  created.  The  paper-and-pencil  survey  process  is  slow  and  labor 
intensive.  It  is  also  not  environmentally  friendly  when  you  consider  all  the  trees  consumed  for 
paper,  optical-scan  sheets  and  envelopes.  Paper-and-pencil  surveys  will  probably  be  used  less 
often  in  the  years  ahead. 


Automated  Surveys 

In  the  age  of  computers  and  the  “information  superhighway,”  automated  surveys  (that  is, 
those  that  respondents  take  on  personal  computers)  are  being  developed  and  used  more 
frequently.  These  waves  of  the  future  can  be  administered  on  disk,  over  E-mail,  or  on  special 
drivers  on  area  networks.  They  can  also  be  administered  over  the  internet.  There  are 
commercially  available  authoring  systems,  for  instance  from  Rayosoft.  Scientists  and  computer 
specialists  with  the  Armstrong  Laboratory’s  Human  Resources  Directorate  and  the  Institute  for 
Job  and  Occupational  Analysis  have  developed  a  windows-based  automated  survey  authoring 
system.  This  technology  will  handle  very  complex  task  and  attitude  survey  requirements.  For 
more  information  on  this  system,  contact  Dr.  Winston  Bennett  at  (210)  536-1981  (The  DSN 
prefix  is  240).  Dr.  Bennett’s  address  is  the  same  as  mine  except  the  organizational  symbol  is 
AL/HRCF.  For  many  applications,  using  an  automated  format  provides  the  advantages  of  easier 
and  faster  administration,  lower  cost,  higher  response  rate,  and  easier  analysis.  Also,  research  at 
the  Armstrong  Laboratory  has  shown  that  most  respondents  prefer  this  format. 

Using  an  automated  format  is  typically  not  too  expensive.  If  respondents  have  easy 
access  to  personal  computers,  it  can  be  an  appealing  option.  This  format  is  also  ideal  for  quick- 
turnaround  studies  where  data  needs  to  be  gathered  and  analyzed  rapidly,  for  instance,  to  make 
quick  decisions  on  critical  issues. 

Telephone  Surveys  and  Computer-Assisted  Telephone  Interviewing  (CATI) 

Another  format  is  the  telephone  survey  which  can  range  from  a  simple  paper-and-pencil 
survey  read  over  the  phone  to  Computer- Assisted  Telephone  Interviewing  (CATI),  which  is  quite 
sophisticated.  It  is  like  an  automated  survey  but  includes  an  interviewer  asking  questions 
prompted  at  a  computer  screen  and  inputting  data  via  a  computer.  Telephone  surveys  are  often 
used  in  marketing  research  and  for  political  polling.  The  inclusion  of  a  skilled  interviewer 
makes  it  nearly  as  personal  as  a  face-to-face  interview,  while  avoiding  the  need  for  interviewers 
to  intrude  physically  into  the  personal  space  (home  or  office)  of  those  being  interviewed. 


4 


However,  like  a  face-to-face  interview,  it  provides  better  opportunity  to  motivate  participation  or 
probe  about  what  has  been  said  than  would  be  possible  with  a  paper-and-pencil  survey. 
Supervisors  can  listen  in  on  interviews  as  they  train  novice  interviewers.  Real-time  veracity 
checks  are  possible  and  data  can  be  created  during  the  interview.  Like  automated  surveys,  it  is  a 
good  format  to  use  for  quick  turnaround  studies.  On  the  down  side,  up-front  costs  can  be  high,  it 
is  labor  intensive,  and  interviewers  must  be  quite  skilled.  CATI,  sometimes  referred  to  using  the 
broader  term  Computer  Assisted  Data  Collection  or  CADAC,  is  discussed  by  Saris  (1991). 

Telephone  surveys  are  often  conducted  by  political  pollsters.  Such  polls,  as  U.S.  News 
and  World  Report  (1995)  documented  disdainfully,  can  be  flawed,  and  elected  officials  may  rely 
too  heavily  on  them  rather  than  on  their  own  inner  vision  and  sense  of  leadership.  However,  if 
carefully  conducted,  they  can  be  a  rich,  rapid  source  of  the  views  of  people  at  the  grass  roots. 

For  interesting  accounts  of  public  opinion  polling  and  its  relationship  to  democratic  processes 
and  politics,  see  Bradbum  and  Sudman  (1988)  and  Fishkin  (1995). 

Do  not  confuse  telephone  interviewing  with  the  dubious  practice  of  having  a  recorded 
message  talk  to  you  when  you  pick  up  the  phone. 


Criteria  for  Selecting  Survey  Type 

Which  survey  format  should  you  use?  Should  you  use  another  method  in  preference  to  a 

survey?  In  making  such  decisions,  consider  the  following  criteria: 

•  Sample  Size.  If  your  sample  is  large,  a  paper-and-pencil  format  will  do,  but  an  automated 
approach  is  often  better.  If  your  sample  is  large  and  geographically  dispersed,  and  you  need 
the  intimacy  of  a  trained  interviewer  to  persuade  prospective  respondents  to  take  the  survey 
(called  a  “conversion  attempt”)  or  to  conduct  in-depth  probes,  consider  CATI.  However, 
beware  that  costs  may  be  high. 

•  Available  Personnel  and  Financial  Resources.  If  you  have  people  to  share  the  tedious 
work  with  you  and  not  much  money,  a  paper-and-pencil  survey  may  be  your  method  of 
choice,  especially  if  the  survey  is  short  and  sample  size  is  small.  However,  consider  an 
automated  survey  approach,  since  it  need  not  be  expensive,  and  you  can  avoid  printing, 
mailing  and  scanning  costs.  If  you  do  not  have  much  money,  forget  CATI. 

•  Time  Pressure.  Do  you  need  to  get  your  survey  out,  back  and  analyzed  fast?  If  so,  consider 
CATI  or  the  automated  survey  approach.  Paper-and-pencil  surveys  are  quite  time 
consuming. 

•  Need  for  Intimacy  and  Flexible  Detail.  Do  you  need  the  questioning  skills  and  social 
bonding  with  respondents  that  a  trained  interviewer  provides?  For  instance,  do  you  need  to 
tease  out  extremely  rich  information  in  a  flexible  way?  If  so,  choose  CATI.  Better  yet, 
choose  face-to-face  interviewing.  This  is  also  a  situation  where  naturalistic  observation  may 
be  suitable. 

•  Availability  of  Automated  Technology.  Do  your  respondents  all  have  good  computers  and 
access  to  a  good  E-mail  system,  a  local  or  wide-area  network,  or  the  internet?  Then  an 
automated  survey  may  be  the  way  to  go. 
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•  Objectionable  or  Intrusive  Questioning.  I  recommend  that  you  avoid  or  limit  the 
inclusion  of  objectionable  or  overly  intrusive  items,  if  possible.  If  you  must  include  such 
items  (they  are  sometimes  legitimate),  ask  for  such  content  in  a  way  that  that  minimizes 
threat.  Also,  provide  maximum  privacy  protection.  The  method  of  choice  in  such  a  situation 
is  difficult  to  say.  An  anonymous  survey  provides  a  safe  distance  between  the  information 
requester  and  the  information  provider  and  ensures  complete  confidentiality.  However,  face- 
to-face  interviews  and  CATI  provide  human  contact  in  which  trust  can  develop  and  sources 
of  concern  can  be  discussed.  Trained  interviewers  also  have  the  skill  to  create  a  non¬ 
threatening  situation  and  get  people  to  open  up.  This  issue  is  addressed  under  Limit 
Objectionable  or  Very  Intrusive  Items  on  page  19  and  Make  Objectionable  Items  More 
Acceptable  on  page  55. 

•  Non-Verbal  Cues  and  Communication.  Are  you  interested  in  body  language?  Do  you 
want  to  see  if  there  is  a  disconnect  between  what  is  being  said  and  what  is  actually  felt?  Do 
you  want  to  tap  underlying  emotional  issues  with  clarity?  Do  you  want  to  observe  social 
interaction  among  participants  and  get  them  to  react  to  the  statements  or  gestures  of  others? 
If  these  issues  are  important,  choose  interviews.  If  you  want  to  observe  all  this  in  a  natural 
setting,  consider  naturalistic  observation. 
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Chapter  3 


Motivating  Participation 

A  survey  developer’s  job  is  to  motivate  participation  as  well  as  to  write  good  items.  You 
need  to  interest  people  in  taking  your  survey,  especially  given  that,  as  discussed  in  the  next 
chapter,  participation  should  be  voluntary.  This  may  be  difficult  since  people  frequently  throw 
surveys  away.  You  are  at  a  disadvantage  relative  to  interviewers  because  you  are  typically  not 
there  in  person  to  persuade  potential  respondents  to  participate.  If  your  survey  is  group- 
administered,  you  are  in  a  better  position  to  encourage  people  to  complete  it.  However,  they 
may  feel  more  pressured  to  participate  and  resent  it. 

Increase  Response  Rate 

Survey  developers  need  to  motivate  participation  because  response  rates  (sometimes 
called  cooperation  rates)  can  be  low.  Sudman  (1983)  reports  that  in  poorly  designed  mail 
surveys,  response  rates  as  low  as  5  or  10  percent  are  sometimes  encountered.  This  is  far  too 
small  a  response  rate  to  trust  the  results  obtained.  If  response  rate  is  very  low,  the  investigator 
will  have  insufficient  cases  to  perform  trustworthy  analyses.  Thus,  the  representativeness  of  the 
sample  (the  extent  to  which  it  accurately  reflects  the  population  you  are  interested  in)  will  be 
questionable. 

People  often  claim  to  be  “surveyed  to  death”  and  state  that  they  Just  do  not  like  surveys. 
People  probably  are  over-surveyed.  This  problem  is  compounded  by  the  fact  that  some  surveys 
are  of  poor  quality,  some  are  too  long,  and  overly  intrusive  questions  are  sometimes  asked.  In 
addition,  people  often  do  not  understand  the  importance  of  a  survey.  Also,  they  may  never  get 
feedback  about  how  their  input  produced  beneficial  change.  These  factors,  and  the  voluntary 
nature  of  surveys,  can  result  in  refusals  to  participate. 

The  situation  is  not  entirely  bleak.  It  is  my  experience  that  with  high  quality  paper-and- 
pencil  surveys,  response  rates  of  between  50  and  70  percent  can  be  expected.  With  computer- 
assisted  telephone  surveys  involving  human  interviewers,  or  with  automated  surveys,  response 
rates  are  typically  higher. 

If  your  response  rate  is  quite  low,  you  can  send  a  follow-up  survey  (to  everyone,  or  to 
those  who  did  not  respond  if  you  have  been  tracking  who  did  respond).  You  can  also  send  a 
follow-up  letter,  post  card  or  E-mail  message  to  encourage  people  to  complete  and  return  their 
survey.  This  could  be  done  automatically  when  an  automated  survey  is  sent  and  returned  via  E- 
mail.  Sometimes  a  follow-up  survey  is  sent  only  to  people  in  an  occupation  or  in  a  location 
where  the  response  rate  was  low. 

Be  proactive.  Take  steps  beforehand  to  motivate  participation  so  the  need  for  follow  up 
is  reduced.  If  you  or  your  points  of  contact  (POCs)  can  administer  the  survey  in  person  via  a 
group  administration,  do  so.  Prepare  your  POCs  to  do  this  task  well.  Use  personal  contact  to 
explain  the  importance  of  the  survey  and  the  voluntary  and  confidential  nature  of  participation. 
Answer  questions  and  allay  fears. 

Group  administration  is  often  not  practical.  Usually,  you  are  not  present  and  need  to  use 
other  means  to  motivate  potential  respondents.  Advance  warning  and  the  offer  of  modest 
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payment  for  participation  are  sometimes  used.  Motivation  can  be  enhanced  by  stressing  the 
importance  of  the  survey  and  the  constructive  change  likely  to  result.  This  can  be  accomplished 
in  an  advance  letter,  in  a  cover  letter,  or  in  introductory  comments.  Also,  respondents  like  to  be 
kept  informed  about  what  was  eventually  done  with  the  information  provided.  This 
demonstrates  they  had  an  impact.  The  professional  appearance  of  your  survey  is  an  indirect  way 
to  show  you  are  serious.  A  cover  letter  can  indicate  you  have  buy-in  from  top  management. 
Easy-to-understand  directions  can  also  help  encourage  participation.  If  your  survey  is  short, 
more  people  will  respond  to  it.  These  issues  are  discussed  below.  In  the  next  chapter, 
confidentiality  and  voluntary  participation  are  discussed.  These  ethical  issues  also  influence 
respondent  motivation. 

Inform  Respondents  Ahead  of  Time 

A  technique  used  by  commercial  firms  is  to  alert  the  respondent  ahead  of  time  that  that 
he  or  she  will  be  receiving  a  survey  by  mail,  phone  or  computer.  This  can  be  done  by  letter,  post 
card,  phone  call.  E-mail  or  even  fax.  Respondents  are  even  sometimes  provided  with  a  1-800 
call-in  number  should  they  have  questions.  This  number  can  also  be  used  for  respondents  to  take 
a  telephone  survey  at  a  time  convenient  to  them.  Advance  warning  puts  the  respondent  in  the 
position  of  expecting  the  survey,  and  gives  you  the  opportunity  to  explain  its  purpose  and 
importance. 

Consider  Paying  for  Participation 

Usually,  respondents  to  organizational  surveys  are  not  paid  for  participation.  Being 
consulted  and  gaining  an  opportunity  to  have  impact  are  sufficient  rewards  for  the  short  time 
spent.  Payment  is  more  likely  in  a  marketing  situation  especially  if  focus  group  interviews  are 
used  instead  of  a  survey.  In  survey  research,  if  payment  or  other  extrinsic  reward  is  provided,  it 
is  usually  modest.  Too  great  a  payment  could  lessen  intrinsic  motivation.  Rather  than  pay 
everyone.  Pious  (1996)  reports  using  a  drawing  to  motivate  participation.  The  winning 
respondent  received  $500.00.  In  a  commercial  situation,  payment  may  consist  of  a  coupon  for  a 
free  meal  at  a  restaurant  or  a  discount  on  the  purchase  of  a  product  or  service.  Using  the 
restaurant  example,  you  would  not  want  to  introduce  bias  by  giving  a  coupon  to  only  one 
restaurant  if  several  restaurants  were  being  evaluated.  Participants  may  be  influenced  to  evaluate 
the  coupon  provider  more  favorably  than  the  competition 

Another  form  of  “payment”  that  I  highly  recommends  is  to  provide  respondents  with 
some  form  of  aggregate  survey  results.  You  can  also  explain  what  was  done  to  make 
improvements  based  on  the  results.  Sieber,  (1992,  p.  40)  is  somewhat  skeptical  of  the  practice  of 
promising  to  make  research  results  available  to  participants.  She  laments  the  promise  may  be 
difficult  to  keep  and  is  often  broken.  She  further  states  that  the  results  of  the  study  may  not  be  of 
interest  to  respondents.  She  recommends  that  subjects  be  debriefed  after  survey  administration. 
However,  this  may  be  difficult  if  the  researcher  is  not  present. 

Ensure  a  Professional  Appearance 

Professional  appearance  is  important,  regardless  of  survey  format.  The  more 
professional  your  survey  looks,  the  more  likely  a  respondent  will  be  to  take  it  since  it  conveys 
that  you  are  competent  and  serious  about  your  survey.  Professional  means  both  attractive  and 
flawless.  Use  the  best  hardware/software  you  can  find  (word  processing  package,  printer. 
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automated  authoring  system,  etc.)  and  pay  attention  to  spelling,  grammar,  proper  spacing,  layout 
and  consistency.  Make  use  of  color,  shading  and  attractive  graphics  if  you  can. 

Add  an  Optional  Cover 

A  cover  is  desirable,  but  not  necessary  if  your  survey  is  short.  If  the  survey  is  an  Air 
Force  survey,  it  will  usually  have  a  Survey  Control  Number  (SCN)  and  expiration  date  (see 
Request  Approvals  and  Exemptions  on  page  65).  The  cover  usually  also  contains  the  survey 
name,  and  the  name  and  address  of  the  responsible  organization. 

Include  a  Cover  Letter 

A  cover  letter  is  often  used  to  introduce  the  survey  to  respondents.  Here,  and/or  in 
introductory  comments,  stress  the  survey’s  importance,  voluntary  participation  and 
confidentiality.  This  is  also  a  good  place  to  mention  you  will  provide  a  results  summary  later,  if 
you  choose  to  do  so.  If  you  use  a  cover  letter,  it  is  best  to  have  it  signed  by  a  high-level  person 
known  to  have  a  legitimate  interest  in  the  survey  results  and  the  power  to  bring  about  needed 
changes.  This  lets  respondents  know  you  have  the  support  of  top  management  (or  other 
influential  clients).  I  have  sometimes  seen  cover  letters  signed  by  the  researcher.  This  is  less 
effective  since  researchers  do  not  convey  the  same  sense  of  institutional  backing  or  authority. 
They  are  also  not  usually  affiliated  with  the  institution  sponsoring  the  research.  After  you  have 
identified  the  dignitary  to  sign  your  letter  (for  example,  the  commander  or  CEO),  do  not  expect 
him  or  her  to  write  it.  Ghost-write  the  letter  for  this  person.  Let  him  or  her  review  the  letter 
mostly  for  style  and  resist  changes  to  its  content.  Having  a  dignitary  sign  your  cover  letter 
(called  “prestige  suggestion”)  increases  the  survey’s  perceived  importance  and  legitimacy, 
increasing  motivation. 

Two  examples  of  cover  letters,  are  provided  in  Appendix  A.  The  first,  written  by  me, 
was  signed  by  the  Air  Force  Surgeon  General  and  was  used  on  the  USAF  Medical  Service 
Objective  Medical  Group  (OMG)  Survey.  The  second  letter  was  signed  by  the  Assistant 
Secretaiy  of  the  Air  Force,  Financial  Management  and  Comptroller,  who  took  a  suggested  letter 
written  which  I  wrote  and  modified  it.  This  letter  was  used  on  the  USAF  Financial 
Management  Career  Survey.  Another  good  source  of  cover  letter  suggestions  is  Sieber  (1992, 
pp.  34-37).  She  discusses  consent  statements,  suggests  what  to  include,  and  provides  a  sample 
letter. 

Provide  Introductory  Comments  and  Instructions 

Introductory  comments  and  instructions  are  advised,  even  if  a  cover  letter  is  used.  If  you 
have  already  covered  most  of  the  important  topics  in  a  cover  letter,  you  can  be  brief  except  for 
general  and  specific  instructions.  If  important  topics  such  as  the  purpose  and  importance  of  the 
survey  and  the  ethical  safeguards  of  voluntary  participation  and  confidentiality  have  not  been 
covered,  do  so  here.  I  often  briefly  repeat  mention  of  voluntary  participation  and  confidentiality 
because  these  issues  are  so  important  and  some  people  ignore  cover  letters.  Also,  generally 
explain  how  to  complete  the  survey.  You  can  always  have  more  specific  instructions 
interspersed  throughout  the  survey.  If  you  use  a  separate  optical  scan  sheet,  explain  how  to  fill  it 
out.  For  example,  you  should  instruct  them  to  use  a  number  2  pencil,  to  mark  their  responses  on 
the  answer  sheet,  to  make  marks  heavy,  to  avoid  stray  marks,  and  to  not  fold,  staple  or  damage 
the  sheet.  Sometimes  an  example  of  the  right  way  and  the  wrong  way  to  mark  the  answer  sheet 
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is  provided.  Embedded  answer  sheets  (that  are  a  part  of  a  paper-and-pencil  survey  itself)  are 
better  than  separate,  general-purpose  scan  sheets.  Let  respondents  know  who  to  contact  if  they 
have  questions.  This  person  could  be  you.  However,  you  way  want  to  shelter  yourself  from  the 
possibility  of  being  contacted  by  many  people,  especially  if  your  sample  is  large.  If  you  have 
others  assume  this  responsibility,  take  care  to  select  and  train  socially  skilled,  empathic  persons, 
aware  of  the  survey  content  and  knowledgeable  enough  to  field  questions.  Sometimes  this  job 
goes  to  people  at  the  respondents’  work  locations  who  are  trained  for  this  role  as  part  of  their  job. 
At  other  times,  the  task  may  go  to  a  point  of  contact  who  has  had  little  experience  with  survey 
administration.  Do  what  you  can  to  prepare  them.  If  you  cannot  train  them  in  person,  you  can 
send  them  a  brief  tutorial  on  the  information  they  need  to  know,  how  to  distribute  or  administer 
the  surveys  and  how  to  handle  incoming  questions.  If  you  have  multiple  contact  persons,  you 
may  need  to  be  vague  and  say  something  like: 


If  you  have  any  questions,  please  contact  your  local  survey  administrator. 


If  one  individual  serves  as  your  contact  person,  provide  his  or  her  name,  address,  phone  number, 
and  possibly  a  fax  number  or  E-mail  address.  An  example  of  introductoiy  comments  and 
instructions  adapted  from  the  San  Antonio  Air  Logistics  Center  Career  Planning  Survey  is 
provided  as  Appendix  B. 

In  your  introductory  comments,  or  in  your  cover  letter,  you  may  encourage  people  to  be 
frank.  However,  never  insist  they  tell  the  truth  or  threaten  punishment  if  they  do  not.  Such 
statements  could  easily  alienate  respondents.  They  could  decide  not  to  complete  your  survey.  In 
addition,  it  would  do  little  good  since,  in  most  instances,  you  could  not  determine  if  they  were 
telling  the  truth.  If  you  could,  it  would  be  costly,  labor  intensive  and  too  intrusive  to  verify  their 
veracity. 

In  some  instances  you  could  encourage  truth  telling  by  indicating  your  intent  to  conduct 
a  veracity  check.  For  instance,  on  an  application  for  employment  (which  is  a  crude  survey),  you 
could  indicate  you  intention  to  contact  former  supervisors  or  teachers.  Such  veracity  checks 
should  be  limited,  not  very  invasive,  and  considered  reasonable  by  the  respondent.  If  you  are 
insensitive  to  the  right  to  privacy  and  indicate  you  will  expend  great  effort  to  assess  veracity, 
your  efforts  will  backfire.  You  will  likely  encourage  deception  or  nonparticipation. 

Limit  the  Number  of  Items 

Keep  your  survey  short.  I  recommend  that  almost  any  survey  should  be  able  to  be 
completed  in  30  minutes  or  less  by  the  t3q)ical  respondent.  In  some  applications,  like  customer 
satisfaction  or  quality  progress,  it  is  probably  easy  to  keep  your  survey  very  short— 25  items  or 
less.  My  longest  survey  was  160  items.  Even  at  that  length,  most  respondents  were  able  to 
complete  it  in  less  than  30  minutes.  However,  fewer  items  are  recommended.  A  good  upper 
limit  is  about  125  items.  Don’t  just  limit  items;  also  limit  the  number  of  scales,  and  the  number 
of  shifts  from  one  scale  to  another.  Some  survey  developers  ask  participants  to  respond  to  the 
same  item  stems  using  many  different  scales.  Sometimes  multiple  scales  for  the  same  item  are 
legitimate,  for  instance  in  probing  for  importance  and  satisfaction  or  how  things  are  versus  how 
they  should  be.  However,  do  not  overdo  it.  If  your  items  are  short,  comprehensible,  and  most 
use  a  single,  common  scale,  then  even  long  surveys  can  be  taken  quickly.  The  number  of  items 
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included  can  be  higher  than  on  surveys  having  longer  items,  frequent  scale  shifts,  items  with 
multiple  scales,  or  many  items  with  item-specific  responses. 

Sometimes  survey  developers  allow  organizations  to  add  specific  items  of  their  own 
choosing,  usually  at  the  end  of  the  survey.  Although  this  practice  adds  length  and  may 
sometimes  result  in  the  inclusion  of  poorly  worded  items,  it  has  the  advantage  of  allowing 
organizations  or  workgroups  to  tailor  the  survey  to  address  their  specific  concerns.  Survey 
participants  may  also  appreciate  the  chance  to  respond  to  organization-specific  items. 

Other  ways  to  motivate  participation  include  ensuring  voluntary  participation, 
maintaining  confidentiality  and  limiting  the  number  of  items  respondents  would  consider 
irrelevant,  offensive  or  too  intrusive.  These  issues  are  discussed  in  the  next  chapter  on  survey 
ethics. 
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Chapter  4 


Survey  Ethics:  Gaining  Important  Information 
While  Protecting  Individuals  From  Harm 

Here  are  just  a  few  reasons  why  surveys  are  conducted: 

•  assess  customer  needs  and  satisfaction, 

•  evaluate  the  success  or  failure  of  organizational  interventions  or  social 
programs, 

•  improve  organizational  or  societal  functioning  by  identifying  problems  which 
need  resolution, 

•  improve  the  quality  or  marketability  of  products  or  services, 

•  understand  the  incidence  and  range  of  social,  political,  spiritual,  or  religious 
attitudes, 

•  determine  how  knowledgeable  the  public  is  about  certain  issues, 

•  identify  the  incidence  of  various  behaviors,  some  of  which  may  be  disapproved 
of  or  illegal, 

•  measure  the  recall  of  or  reaction  to  felt  or  displayed  emotions, 

•  measure  personality, 

•  anticipate  or  understand  respondent  reactions  to  issues  or  events, 

•  evaluate  individuals  for  employment,  promotion  and  other  purposes,  and 

•  facilitate  grass-roots  participation  in  the  political  process  through  “tovra  hall” 
meetings  and  public  polling. 

Surveys  are  by  their  nature  intrusive.  Information  about  attitudes,  beliefs,  interests,  values, 
behaviors  and  background  is  collected  which  at  the  level  of  the  individual  may  be  personal  and 
which  at  any  level  may  sometimes  be  controversial.  If  some  of  this  information  were  linked  to 
specific  individuals,  it  could  put  them  in  jeopardy  of  harassment,  social  ostracism,  denial  of 
opportunities,  and  even  incarceration.  Thus,  safeguards  must  be  taken  to  protect  individuals  who 
provide  such  information  from  harm. 

In  the  discussion  below  and  in  Appendices  C,  D,  and  E,  I  give  a  detailed  account  of 
ethical  considerations  in  interview  or  survey  work.  For  other  detailed  accounts  applicable  to  all 
forms  of  research  with  human  subjects,  see  Kimmel  (1988)  and  Sieber  (1992).  Older  interview 
and  survey  texts  frequently  failed  to  discuss  ethical  issues.  Newer  ones  usually  give  some 
attention  to  ethics  and  there  is  strong  agreement  that  interviews  and  surveys  should  be  voluntary 
and  privacy  should  be  protected.  Thus,  what  I  say  in  greater  detail  here  is  consistent  with  the 
recent  writings  of  other  survey  and  interview  experts  (see  Fink  &  Josecoff,  1985;  Fowler,  1993; 
and  Salant  &  Dillman  1994)  and  other  applied  social  researchers  (Kimmel,  1988;  Sieber,  1992). 

The  Rights  Perspective:  Facilitating  Free  Speech  by  Protecting  Privacy  and  Ensuring 
Confidentiality 

The  ethical  principles  which  guide  survey  data  gathering  are  rooted  in  two  basic  human 
rights:  free  speech  and  privacy.  More  specifically,  survey  participants  have  the  right  to: 

•  speak  freely,  without  constraint,  even  if  others  may  not  like  what  they  say,  and 
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•  remain  silent,  or  if  they  speak,  to  set  limits  on  the  personal  information  they  divulge,  and 
have  what  they  say  as  individual  information  providers  remain  confidential  (unless  they 
consent  to  disclosure). 

Survey  professionals  should  encourage  unconstrained  speech  to  gain  a  thorough 
understanding  of  attitudes,  beliefs,  values  and  behaviors.  When  speech  becomes  so 
unconstrained,  and  since  surveys  are  intrusive,  the  privacy  of  respondents  should  be  protected 
and  confidentiality  should  be  guaranteed  to  prevent  harm.  If  information  about  specific 
individuals  is  disclosed,  it  should  be  with  their  consent. 

Potential  respondents  should  be  given  the  right  not  to  participate.  Survey  professionals 
motivate  voluntary  participation  by  creating  a  “safe  haven”  so  that  most  potential  respondents 
will  feel  comfortable  enough  to  participate.  Hopefully,  potential  participants  will  want  to 
respond  (not  just  feel  comfortable  doing  so)  if  the  survey  developer  successfully  conveys  the 
importance  of  the  survey  and  explains  the  positive  outcomes  that  are  likely  to  accrue. 

The  rights  to  free  speech  and  to  privacy  that  are  the  foundation  of  the  author’s  ethical 
perspective  have  their  roots  in  the  amendments  to  the  US  Constitution,  various  state 
constitutions,  and  in  American  common  law.  The  rights  perspective,  which  may  only  be  of 
interest  to  some  readers,  is  discussed  in  Appendix  C. 

We  need  not  only  look  to  the  Constitution,  American  law,  or  the  rights  perspective  to 
find  ethical  principles  to  guide  survey  professionals.  Professional  ethical  codes  also  provide 
sound  guidance. 

Professional  Ethics:  The  Data  Gatherer’s  Responsibility  to  the  Information  Provider 

Ethical  implications  for  conducting  surveys  and  protecting  individual  respondents  can  be 
drawn  from  professional  guidelines.  Survey  developers  come  from  diverse  professional  and 
technical  backgrounds.  Most  professions,  through  professional  associations,  specify  ethical 
principles  to  guide  professional  behavior.  Ethical  guidelines  exist  for  psychologists,  social 
workers,  physicians,  lawyers  and  many  other  professionals.  Despite  this,  there  are  opinion 
research  associations  that  provide  disclosure  rules  and  codes  for  ethical  conduct.  For  instance. 
The  National  Council  on  Public  Polls  and  the  American  Association  of  Public  Opinion  Research 
have  such  guidelines  (Fink  &  Kosecoff,  1985,  p.  51).  The  ethical  principles  guiding  different 
professionals  are  usually  similar.  The  predominant  messages  are  clear.  Respect  peoples’  rights 
and  dignity.  Help  rather  than  harm  your  clients.  Protect  the  privacy  of  individuals  who  provide 
you  with  information.  Keep  what  they  tell  you  confidential:  in  all  but  the  rarest  situations  do  not 
release  information  about  individuals.  If  you  must  release  such  information,  do  so  only  after 
receiving  their  consent.  Confidentiality  and  preventing  harm  are  such  core  ethical  principles  that 
Kimmel  (1988)  reports  they  appear  in  all  American  and  European  ethical  codes  for 
psychological  research. 

Professional  ethical  guidelines  are  based  in  part  on  two  key  legal  and  ethical  concepts: 
privilege  and  informed  consent.  Privilege  refers  to  the  (usually)  inviolate  confidentiality  of 
communications  between  certain  professionals  (lawyers,  doctors,  clerics,  mental  health 
professionals)  and  their  clients  and  the  right  of  testimonial  privilege  in  courts  of  law.  Privilege  is 
a  well  established  principle  in  the  civilian  community.  However,  it  is  less  well  recognized  in  the 
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Federal  Government,  including  the  Department  of  Defense  (DoD),  as  discussed  in  the 
shadowbox  below. 


The  Privacy  Act  and  the  Limits  of  Privacy  Protection  in  Federal  Facilities.  A  Wall 
Street  Journal  article  by  Pollock  (1996)  raised  my  consciousness  about  the  limits  on  privilege  in 
Federal  facilities.  Pollock  described  an  Air  Force  mother’s  unsuccessful  attempt  to  keep  the 
notes  of  an  Air  Force  psychiatrist  about  her  daughter’s  alleged  rape  out  of  the  hands  of  lawyers 
for  the  accused-notes  obtained  by  Air  Force  investigators  against  the  will  of  the  parties 
involved.  Pollock  consulted  a  Pentagon  lawyer  for  clarification.  He  explained  that  the  armed 
forces  do  not  recognize  privilege  since  all  medical  and  mental  health  records  on  a  base  belong  to 
the  Government  and  can  be  examined  by  military  officers  in  the  performance  of  their  duties. 
Checking  further  into  this,  I  obtained  an  informal  document  used  by  medical  clinic  practitioners 
at  Brooks  AFB  titled  “Legal  Issues  Affecting  Medical  Administration”  in  which  it  is  stated  that 
the  “Federal  Privacy  Act  allows  access  to  patient  records  (without  patient  consent)  by  persons  in 
the  Department  of  Defense  who  have  a  legitimate  duty  purpose  for  the  information”  including 
law  enforcement  officials,  trial  counsel,  commanders,  federal  investigators  and  congressional 
staffers.  The  text  of  the  Privacy  Act  of  1974  (Public  Law  93-579,  United  States  Code,  Title  5, 
Part  1,  Chapter  5,  Subchapter  II,  Section  552a),  confirms  this.  This  act  can  be  accessed  on  the 
internet  and  is  also  included  as  appendix  B  in  Bushkin  &  Schaen  (1976).  Although  this  act  was 
designed  in  part  to  protect  people  from  having  records  about  them  disseminated  without  their 
consent,  it  contains  many  exceptions  that  make  it  relatively  easy  for  federal  officials  to  obtain 
personal  information  from  records  in  federal  facilities  without  the  consent  of  the  person 
involved.  In  addition.  Air  Force  Instruction  41-210,  Patient  Administration  Function 
(Attachment  3:  Releasing  Medical  Information),  states  that  Federal  investigative  agents  are  to  be 
granted  access  to  medical  records  (in  an  Air  Force  facility)  upon  proper  identification  and  can 
seize  records  if  they  leave  a  copy.  Related  to  this  issue,  see  the  comments  of  McGuire  on  page 
16  about  being  careful  what  you  write  down  and  disclose  if  you  work  in  a  Federal  facility. 


Informed  consent  refers  to  the  necessity  of  providing  clients  with  an  understanding— in 
language  they  can  easily  comprehend— that  participation  with  a  practitioner  or  researcher  is 
voluntary,  informing  them  of  the  anticipated  consequences  of  participation  or  nonparticipation 
and  the  factors  which  could  reasonably  influence  their  decision,  protecting  prospective 
participants  from  harm  if  they  decline  to  participate,  explaining  other  aspects  about  which  they 
may  inquire,  and  letting  them  know  about  the  limits  of  privacy,  if  any.  As  discussed  in  the 
shadowboxes  below,  there  could  be  conflict  between  professional  associations  and  the  DoD  on 
this  issue  as  well. 

Professional  ethical  guidelines  focus  most  attention  on  the  ethics  of  the  professional’s 
relationship  with  clients.  Although  client  is  often  referred  to  in  this  text  as  the  person  or 
organization  who  requested  the  survey,  each  survey  respondent  is  also  a  client.  Survey 
researchers  must  behave  ethically  in  their  relationships  with  both  types  of  clients. 

Like  physicians  and  others,  survey  professionals  protect  individual  data  from  disclosure. 
They  do  so  by  protecting  the  identity  of  individual  participants  and  by  reporting  aggregated 
(grouped)  data.  If  individual  data  (such  as  verbatim  quotations)  are  reported,  the  identity  of  the 
information  source  is  withheld  unless  he  or  she  has  given  prior  consent.  However,  unlike  priests, 
health  professionals  and  lawyers,  survey  researchers  do  not  have  automatic  testimonial  privilege 
in  courts  of  law.  Subpoenas  can  be  used  to  force  disclosure  of  confidential  research  information. 
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In  1988,  Congress  provided  relief  in  a  law  allowing  researchers  to  seek  a  certificate  of 
confidentiality  to  protect  their  data  from  a  subpoena  (Bradbum  &  Sudman,  1988,  p  200;  Sieber, 
1992,  pp.  59-60).  As  Bradbum  and  Sudman  (1988)  and  Fowler  (1993)  indicate,  researchers  have 
sent  their  data  abroad  to  prevent  breaches  of  confidentiality  in  United  States  courts.  The 
American  Psychological  Association  (APA)  encourages  members  to  familiarize  themselves  with 
steps  to  protect  data  from  disclosure  short  of  noncompliance  with  the  law.  The  APA  has 
prepared  a  document  on  strategies  for  coping  with  subpoenas  or  compelled  testimony  which  is 
available  from  their  Office  of  Legal  Affairs  (American  Psychological  Association,  1995,  1996). 

I  am  most  familiar  with  the  American  Psychological  Association’s  Ethical  Principles  of 
Psychologists  and  Code  of  Conduct  (American  Psychological  Association  or  APA,  1992; 
hereafter  referred  to  as  the  Ethics  Code).  I  reviewed  these  ethical  guidelines  and  an  APA  book  on 
the  Ethics  Code  (Canter,  Bennett,  Jones  &  Nagy,  1994),  extracting  principles  and  standards 
relevant  to  survey  developers.  The  portion  of  the  Ethics  Code  pertaining  to  survey  work  is 
included  as  Appendix  D. 


Conflict  Between  the  APA  Ethics  Code  and  Societal  or  Organizational  Expectations 

Ethical  principles  are  sometimes  in  conflict  with  the  law  or  with  what  is  expected  of 
people  in  their  organizations.  The  Ethics  Code  is  somewhat  vague  on  how  to  deal  with  conflict 
with  the  law,  stating  that  if  ethical  responsibilities  conflict  with  the  law,  professionals  should 
make  clear  their  commitment  to  the  Ethics  Code  and  take  steps  to  resolve  the  conflict  in  a 
responsible  manner  (Ethical  Standard  1 .02).  The  precise  nature  of  the  action  to  be  taken  is  not 
clear,  but  there  is  an  implied  moral  superiority  of  the  Ethics  Code  and  the  recommendation  that 
the  Ethics  Code  not  be  violated.  In  discussing  this  standard.  Canter  et  al.  (1994)  state  that  the 
standard  “does  not  always  require  conformity  with  the  law”  (p.  33).  The  professional  is  also 
encouraged  to  exercise  free  speech  and  clearly  state  his  or  her  ethical  views.  According  to 
Canter  et  al.,  this  could  be  in  the  form  of  a  letter  or  memo  to  one’s  employer  or  supervisor. 

The  APA’s  stand  on  conflict  between  the  Ethics  Code  and  organizational  demands  is 
clearer.  If  organizational  demands  conflict  with  the  Ethics  Code,  the  professional  is  expected  to 
clarify  the  nature  of  the  conflict,  make  known  his  or  her  commitment  to  the  code,  and  resolve  the 
conflict  with  adherence  to  the  Ethics  Code  (Ethical  Standard  8.03).  In  discussing  ethical 
standards  1.02  and  8.03,  Canter  et  al.  (1994)  indicate  that  professionals  are  expected  to  be 
proactive  in  resolving  conflicts.  They  claim  that  failing  to  do  so  is  an  ethical  violation  in  itself. 


Conflict  Between  the  APA  Ethics  Code  and  DoD  Directives 

The  APA  ethics  committee  report  for  1993  (American  Psychological  Association,  1994) 
acknowledges  that  military  psychologists  frequently  face  conflict  between  the  Ethics  Code’s 
confidentiality  provisions  and  the  DoD’s  directives  to  provide  information  without  consent.  The 
ethics  committee  refers  to  ethical  standards  1 .02  and  8.03  (see  discussion  in  the  shadowbox 
above  and  Appendix  D)  which  acknowledge  that  conflicts  can  arise  between  ethical  standards 
and  laws  or  organizational  practices.  The  ethics  committee  states  that  these  standards  “establish 
an  affirmative  duty  of  the  psychologist  (or  other  practitioner)  to  address  the  conflict,  not  just  to 
follow  the  law  or  institutional  rule  without  looking  for  alternatives”  (p.  665).  They  state  that  if 
practitioners  cannot  guarantee  confidentiality,  clients  should  be  informed  of  confidentiality 
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limits  at  the  beginning  of  professional  contacts.  In  addition,  they  recommend  that  practitioners 
be  proactive  in  dealing  with  their  organization  about  their  ethical  misgivings.  For  instance,  they 
could  discuss  their  concerns  about  confidentiality  with  their  commander  when  they  are  initially 
assigned  to  a  clinical  position.  Canter  et  al.  (1994)  acknowledge  that  individual  professionals 
may  have  limited  power  to  affect  organiscational  policies  and  practices.  However,  if  pressured  to 
behave  unethically,  they  recommend  professionals  make  their  objections  know,  explain  code 
requirement,  and  suggest  solutions  without  placing  too  much  burden  on  the  organization. 

In  discussing  the  limits  of  practitioner-patient  confidentiality  in  the  DoD,  McGuire 
(1995/1996)  explains  that  he  always  responded  as  a  psychologist  rather  than  as  a  Naval  officer. 
McGuire  (p.  4)  describes  his  decision  rule  as  simple:  “if  the  confidential  information  does  not ... 
directly  threaten  the  welfare  of  the  nation  or  other  members  of  society,  then  I  maintained 
confidentiality,  including  what  I  chose  to  put  into  the  official  medical  record.  This  is  essentially 
the  same  rule  now  supported  by  the  Ethics  Code  of  the  APA.”  He  also  advises  young  military 
clinicians  to  “know  when  to  just  plain  shut  up!”  Recognizing  that  looking  after  others  is  as 
important  as  looking  after  oneself,  he  goes  on  to  say  “The  expression  ‘Don’t  make  waves’ 
doesn’t  just  apply  to  your  personal  welfare;  it  often  applies  to  those  servicemen  and  women  and 
their  families  who  come  to  you  for  help  and  not  judgment.”  Anderson  (1995/1996)  supports 
McGuire’s  position  and  explains  that  military  personnel  cannot  be  held  accountable  for  failure  to 
report  noncriminal  conduct  that  may  be  reported  to  them  in  a  clinical  interview.  He  refers  to 
statements  about  homosexuality  and  explains  that  health  care  providers  are  not  legally 
accountable  for  failure  to  report  statements  made  about  sexual  orientation  or  behavior  that  the 
military  happens  to  disapprove  of.  This  medical  example  also  applies  to  military  survey  or 
nonclinical  interview  professionals  to  whom  similar  information  may  be  divulged.  Of  course,  it 
applies  to  a  broader  range  of  issues  that  just  sexual  orientation. 


Ethical  decisions  ultimately  reside  with  the  individual.  You  have  the  right  to 
conscientiously  object  to  institutional  or  societal  practices  you  consider  unethical.  If  you  are 
sincere  and  express  your  objection  with  logic,  determination  and  politeness,  you  may  suffer  no 
adverse  consequences.  However,  be  prepared  to  accept  adverse  consequences,  and  to  defend 
yourself  against  such  actions  through  peaceful,  legal  means. 

Your  Ethical  Obligations  to  Survey  Participants 

What  do  ethical  concerns  mean  in  practice?  What  do  you  do  to  ensure  your  survey 
procedures  are  ethical?  In  this  section,  communicating  your  ethical  sensitivity,  ensuring 
voluntary  participation  and  safeguarding  confidentiality  are  discussed.  The  issues  of  limiting 
excessively  intrusive  content  and  eliminating  illegal  content  are  also  discussed. 

Ensure  Voluntary  Participation:  The  right  to  privacy  and  the  right  to  avoid  self 
incrimination  usually  give  people  the  right  to  refuse  to  be  surveyed,  or  to  not  answer  specific 
items.  Bradbum  and  Sudman  (1988,  p.  201)  state  that  all  professional  survey  organizations 
recognize  this  right.  I  strongly  recommends  that  voluntary  participation  be  ensured  and  coercion 
be  avoided.  Only  friendly  persuasion  should  be  used. 


An  Air  Force  Exception  and  Example  of  the  Benefit  of  Voluntary  Participation.  Despite 
the  contention  by  Bradbum  and  Sudman  (1988)  that  the  principle  of  voluntary  participation  is 
universally  recognized,  some  surveys  have  been  mandatory,  and  some  still  are.  Also,  despite  the 
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author’s  recommendation,  some  people  will  disagree,  or  perhaps  not  give  the  issue  much 
thought. 

Generally  in  the  Air  Force,  survey  participation  is  voluntary,  as  recommended  in  this 
text.  However,  this  is  not  the  case  with  the  job  inventories  developed  by  the  Air  Force 
Occupational  Measurement  Squadron  (AFOMS).  The  privacy  act  statement  contained  in  these 
inventories  indicates  that  disclosure  is  mandatory  and  even  threatens  punishment  under 
provisions  of  the  Uniform  Code  of  Military  Justice  (UCMJ)  for  failure  to  complete  the  inventory. 
In  defense  of  the  job  inventories  and  AFOMS,  the  content  is  not  threatening:  respondents  are 
only  asked  to  indicate  the  tasks  they  perform  in  their  jobs  and  estimate  how  much  time  they 
spend  on  them.  These  inventories  do  not  appear  to  contain  information  that  people  would  be 
reluctant  to  provide. 

In  a  recent  telephone  conversation  with  a  representative  of  the  Quality  Office  at  HQ 
ACC  (Air  Combat  Command),  I  mentioned  my  objection  to  mandatory  surveys.  The  person  at 
HQ  ACC  provided  practical  information  to  support  this  position.  He  indicated  that  his 
organization  had  used  a  mandatory  automated  survey  but  decided  to  switch  to  a  voluntary  one. 
When  the  survey  became  voluntary,  they  experienced  a  dramatic  decline  in  the  incidence  of 
respondents  just  holding  down  the  key  that  allowed  them  to  go  mindlessly  through  the  survey 
without  providing  useful  data. 

The  automated  survey  technology  developed  by  the  Armstrong  Laboratory’s  Human 
Resources  Directorate  and  the  Institute  for  Job  and  Occupational  Analysis  contains  the  option  of 
making  an  item  mandatory.  That  is,  the  respondent  must  respond  to  the  item  to  continue.  This 
option  can  be  used  to  encourage  a  response  to  items  the  survey  developer  considers  critical  for 
tracking  or  analysis,  such  as  name  or  social  security  number. 


I  base  my  prohibition  against  mandatory  participation  partly  on  ethical  considerations. 
However,  this  position  is  also  based  on  practical  concerns.  Requiring  people  to  participate,  or 
placing  pressure  on  them  to  respond  to  specific  questions,  is  likely  to  result  in  untrustworthy 
data,  or  no  data.  This  is  especially  true  if  people  are  asked  about  engaging  in  illegal  or  other 
disapproved  activities.  Sieber  (1992)  cautions  that  respondents  should  not  be  asked  directly  if 
they  engaged  in  illegal  behavior.  If  they  had,  they  would  be  unlikely  to  answer  honestly  since 
they  might  place  themselves  in  legal  jeopardy.  It  is  not  only  questions  about  illegal  or 
disparaged  activities  that  could  lead  to  dishonest  or  distorted  responses.  If  persons  are  forced  to 
participate  and  they  are  uncomfortable  with  a  survey’s  content,  they  may  give  misleading 
responses.  As  Sieber  (1992,  p.  7)  says,  “...those  who  cannot  refuse  to  participate  have  a  secret 
weapon  available  to  them  for  the  protection  of  their  privacy  and  autonomy—they  can  lie.” 
Respondents  may  also  provide  random  or  bizarre  response  patterns  without  even  reading  the 
items.  If  they  take  the  survey  seriously,  they  may  simply  skip  objectionable  content. 

As  mentioned  previously,  the  practice  of  ensuring  voluntary  participation  is  based  on  the 
principle  of  informed  consent.  However,  participants  usually  need  not  sign  a  consent  form,  since 
as  Sieber  (1992)  indicates,  respondents  can  refuse  behaviorally.  For  example,  they  can  hang  up 
the  phone,  walk  out  of  the  room,  respond  randomly,  or  throw  a  survey  away.  Also,  participation 
under  voluntary  conditions  implies  consent.  However,  Fowler  (1993)  states  that  consent  forms 
may  be  appropriate  if  participants  are  asked  to  provide  information,  which  if  misused,  could  hurt 
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them.  The  consent  form  would  provide  a  record  that  the  participant  was  informed  about  the 
potential  for  harm  and  the  limits  on  privacy. 

Maintain  Confidentiality  and  Non- Attribution:  As  Seiber  (1992)  and  Salant  and 
Dillman  (1994)  point  out,  there  is  a  distinction  between  anonymity  and  confidentiality. 
Anonymity  is  best,  but  cannot  always  be  promised.  There  is  a  subtle  “can  not”  versus  “will  not” 
distinction.  If  a  survey  is  anonymous,  no  information  is  collected  which  could  link  data  to 
individual  respondents.  Thus,  a  breach  of  privacy  need  not  be  feared.  However,  it  is  the  author’s 
experience  that  many  people  do  not  trust  that  anonymous  surveys  are,  in  fact,  anonymous.  Many 
people  believe  survey  developers  have  some  trick  way  to  identify  them.  Perhaps  this  distrust  is 
understandable  because  survey  researchers  often  can  link  data  back  to  individual  respondents  or 
small  groups.  They  can  do  so  directly  via  an  assigned  code  or  social  security  number,  or 
indirectly  via  demographic  (background)  information.  Respondents  are  often  savvy  enough  to 
realize  you  can  identify  them  through  background  data,  and  even  grouped  data  can  be  used  to 
identify  them.  For  instance,  during  data  analysis,  you  may  discover  a  subgroup’s  “cell  size”  is 
only  one  or  two  people.  Do  not  claim  that  respondents  can  only  be  identified  through  their 
name,  social  security  number  or  survey  code  number.  If  confidentiality  is  promised,  researchers 
imply  or  admit  they  can  tie  individuals  to  responses,  but  promise  they  will  not.  Since  some 
people  don’t  even  trust  anonymous  surveys,  it  has  been  my  approach  to  ignore  the 
anonymity/confidentiality  distinction  and  to  not  argue  whether  respondents  can  be  identified 
unless  prospective  participants  bring  the  issue  up.  Rather,  I  stress  confidentiality  and  promise 
that  individual  data,  except  for  verbatim  comments  without  attribution,  will  not  be  released. 

As  discussed  in  Appendix  D,  survey  professionals  have  an  obligation  not  to  harm 
respondents  or  diminish  their  dignity.  Improperly  protected  survey  data  could  do  both.  For 
example,  in  organizational  surveys,  respondents  may  express  animosity  toward  or  distrust  of 
organizational  leaders.  This  could  incur  the  wrath  of  these  people  and  retaliation  from  them.  In 
other  surveys,  they  may  report  behaviors  which  some  people  consider  immoral,  confess  to  illegal 
behaviors,  or  express  social,  political  or  religious  views  that  are  not  shared  by  most  people.  In 
the  wrong  hands,  such  data  linked  to  individuals  would  be  damaging.  Survey  researchers 
determine  the  incidence  of  attitudes  or  behaviors  in  groups  of  individuals.  They  examine 
aggregate  rather  than  individual  data.  They  use  this  information  to  better  understand  issues,  to 
improve  policies  and  practices  and  to  help  people  rather  than  harm  them. 

If  you  know  of  institutional  or  legal  constraints  on  confidentiality  (for  instance, 
company  or  court  directives  forcing  disclosure  in  some  instances),  mention  these  to  participants 
up  front  so  they  can  make  an  informed  decision  about  participation. 

The  concern  about  confidentiality  is  reasonable  due  to  the  possible  misuse  of  data  and 
the  potential  for  harm.  Confidentiality  creates  the  condition  where  sensitive  information  can  be 
shared  without  threat  if  respondents  trust  it  will  be  maintained.  To  gain  respondent  cooperation, 
explain  in  a  cover  letter,  or  in  introductory  comments,  how  survey  data  will  be  used,  how  privacy 
will  be  safeguarded,  and  the  steps  you  have  taken  (or  will  take)  to  prevent  misuse.  Stress  that 
data  will  be  reported  to  management  or  other  clients  only  in  aggregate  form.  If  your  survey  is  an 
organizational  survey,  indicate  that  individual  responses  will  not  be  shared  with  supervisors  or 
anyone  else  in  the  chain  of  command,  except  individual  comments  which  may  be  transcribed 
verbatim.  Explain  that  even  when  individual  comments  are  shared  (for  instance,  in  verbatim 
quotes),  by-name  references  will  be  removed  except  when  the  reference  is  obvious,  such  as  to  a 
commander  or  CEO. 
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If  possible,  avoid  asking  for  name  or  social  security  number  (SSN).  If  such  information 
must  be  collected  (for  instance  to  conduct  longitudinal  research  or  to  match  data  files),  assign 
random  codes  and  substitute  them  for  identifying  information  in  your  data  file  immediately. 
Share  the  correspondence  between  people  and  codes  with  as  few  people  on  the  survey  team  as 
possible,  and  only  with  those  you  can  trust  not  to  divulge  this  information.  Limit  access  to  your 
raw  data.  Stress  with  team  members  that  individual  response  data  and  by-name  comment  data 
must  not  be  divulged. 

Infrequently,  clients  may  pressure  you  to  reveal  the  responses  of  specific  individuals.  If 
this  happens,  refuse  such  requests  without  exception. 

Certain  combinations  of  demographic  (background)  data  may  serve  to  identify  specific 
individuals  through  what  Seiber  (1992)  calls  “deductive  disclosure.”  For  instance,  a  person  may 
be  the  only  female  General  Manager- 14  of  Hispanic  heritage  in  an  organizational  subunit  and  the 
researcher  or  survey  client  can  deduce  who  she  is  from  her  demographic  responses.  When  you 
do  subgroup  analyses,  make  sure  that  the  responses  of  individuals  are  not  obvious.  You  do  this 
by  keeping  your  aggregations  large,  or  by  not  reporting  data  in  smaller  cells. 

Confidentiality  protection  need  not  only  pertain  to  individuals.  As  Bradbum  and 
Sudman  (1988,  p.  202)  report,  many  social  scientists  believe  that  the  confidentiality  protection 
should  extend  beyond  individuals  to  organizations.  Thus,  for  instance,  data  obtained  from  a 
specific  organization  would  be  combined  with  data  from  similar  organizations  and  reported  by 
organization  type-say  “Ivy  League  colleges”  or  “environmental  groups.”  Likewise,  data  from 
organizational  subunits  within  a  large  organization  would  not  be  shared  with  other  subunits. 
Individual  subunits  may  be  contrasted,  but  to  avoid  counterproductive  comparisons  by  subunit 
managers  or  employees,  these  data  may  be  held  at  higher  management  levels.  Subunits  could 
also  be  contrasted  with  the  organization  as  a  whole  or  with  other  larger  aggregations  (e.g.  all 
manufacturing  units)  rather  than  with  other  subunits. 

Limit  Objectionable  or  Very  Intrusive  Items:  I  recommend  that  you  limit,  but  not 
necessarily  eliminate,  potentially  objectionable,  threatening,  or  very  intrusive  questioning.  Such 
items  may  be  legitimate  on  some  surveys  since  is  it  important  to  study  controversial  issues  to 
deal  with  them  effectively.  Arvey  and  Sackett  (1993)  caution  that  requesting  information  which 
invades  privacy  will  be  perceived  as  less  fair  in  a  selection  situation  (e.g.,  on  an  application 
form)  than  information  which  is  not  so  invasive.  Regardless  of  the  purpose  of  your  survey,  your 
ethical  obligation  to  maintain  confidentiality  and  protect  people  from  harm  becomes  especially 
critical  when  you  include  very  intrusive  content. 

If  you  must  include  such  content,  ask  yourself  if  you  really  need  to  know  the  information 
asked.  Be  sensitive  in  how  you  ask  for  such  information.  Often,  it  is  wise  to  ask  for  it  indirectly 
(see  Make  Objectionable  Items  More  Acceptable  on  55).  If  such  items  are  included,  emphasize 
your  commitment  to  confidentiality  and  do  not  betray  that  pledge.  Do  what  you  can  to  develop 
trust.  Much  like  the  “there  are  no  right  or  wrong  answers”  statement  we  often  see,  you  can 
explain  in  an  introductory  statement  that  your  intent  is  not  to  judge  the  correctness  of  expressed 
attitudes  or  the  morality/legality  of  reported  behaviors.  Explain  that  you  are  trying  to  determine 
the  prevalence  of  attitudes  or  behaviors  and  have  no  interest  in  linking  them  to  specific 
individuals  for  punitive  purposes.  On  a  survey,  people  can  easily  skip  items  they  consider 
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objectionable  or  intrusive.  Usually  this  is  not  explicitly  stated,  but  discussion  of  voluntary 
participation  implies  it. 

Illustrative  Examples  of  Intrusive  Content.  What  topics  are  particularly  threatening? 
Bradbum,  Sudman  and  associates  (1979,  p.  68)  queried  nearly  1200  interview  participants 
indirectly  about  their  uneasiness  concerning  questions  they  had  just  been  asked.  They  did  so  by 
phrasing  the  exit  questioning  in  terms  of  making  “most  people”  (rather  than  themselves)  feel 
very  uneasy.  Their  list  is  illustrative,  not  exhaustive.  At  the  top  of  the  “very  uneasy”  list  was 
masturbation  (56%)  followed  by  marijuana  use  and  sexual  intercourse  (42%).  A  sizable 
percentage  of  people  were  judged  to  be  very  uneasy  about  questions  concerning  stimulant  and 
depressant  drug  use  and  intoxication  (31%  and  29%  respectively).  Ten  to  13  percent  of  people 
were  judged  to  be  very  uneasy  about  questions  concerning  drinking  alcohol,  gambling  with 
friends  and  revealing  their  income.  Other  leisure  time,  social  and  sports  activities  were  hardly 
threatening.  Respondents  believed  that  only  12  to  21  percent  of  the  population  would  not  be 
threatened  at  all  by  questions  concerning  sexual  activity,  illicit  drug  use,  or  intoxication. 

Despite  feelings  of  uneasiness,  most  questions  were  not  too  personal  for  respondents 
themselves  to  answer.  Interviewees  seldom  refused  to  answer  even  the  most  threatening 
questions.  Only  the  sexual  questions  (intercourse,  masturbation)  were  considered  too  personal 
for  them,  with  34  percent  indicating  this  to  be  the  case.  However,  only  about  seven  percent  of 
the  sample  chose  not  to  answer  these  questions  despite  the  fact  the  interviewers  stressed  up  front 
that  some  questions  might  make  them  uneasy  and  they  did  not  have  to  answer  them.  Bradbum, 
Sudman,  and  their  associates  concluded  from  this  disparity  that  instead  of  refusing  to  answer, 
participants  sometimes  reported  not  engaging  in  threatening  activities  that  they  had  engaged  in. 
This  allowed  them  to  be  appear  to  be  good  respondents  while  projecting  a  socially  desirable 
image. 

The  apparent  willingness  of  people  to  answer  threatening  questions  deceptively  in  an 
interview  brings  up  the  following  points  of  comparison  between  surveying  and  interviewing: 

1)  In  face-to-face  interviews,  more  than  on  surveys,  data  gatherers  know  the  identity  of 
participants.  Since  information  providers  cannot  remain  anonymous  and  interviewers  can  tie 
responses  to  respondents,  participants  may  feel  more  threatened  than  they  would  in  answering  a 
survey.  If  they  admitted  to  holding  attitudes  or  engaging  in  behaviors  that  the  interviewer  or 
others  may  disapprove  of,  they  could  be  ostracized  or  otherwise  harmed.  Answering  deceptively 
gives  the  interviewee  a  benign  way  to  avoid  possible  negative  consequences.  2)  It  is  easy  not  to 
respond  to  objectionable  or  intrusive  items  on  a  voluntary,  anonymous  survey.  Respondents  are 
told  the  survey  is  voluntary.  By  implication,  individual  items  are  voluntary,  and  the  survey 
developer  is  usually  not  present  to  disapprove  of  skipping  them.  It  is  more  difficult  to  say  no  to 
a  persuasive,  persistent  interviewer.  Doing  so  may  be  rude  or  socially  inappropriate.  At  worst,  a 
person  could  get  into  trouble  or  be  denied  employment  or  some  other  opportunity.  Thus,  an 
interviewee  could  experience  enormous  social  pressure  to  respond.  3)  The  interview  creates  a 
social  relationship.  The  participant  may  like  and  want  to  please  the  interviewer.  He  or  she  may 
want  to  be  respected  by  the  interviewer.  This  could  result  in  heightened  social  desirability  bias 
in  an  interview-regardless  of  whether  a  question  is  threatening.  4)  There  is  no  good,  cost 
effective  way  to  check  the  veracity  of  responses  to  interview  questions.  Interviewees  can, 
therefore,  lie  or  distort  their  responses  with  impunity.  Survey  respondents  can  deny  attitudes  or 
behaviors  also,  but  would  feel  less  need  to  do  so.  Of  course,  some  deceptive  responses  may  be 
motivated  more  by  denial  than  by  a  willful  intent  to  deceive. 
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A  recent  examination  of  threatening  content  was  conducted  by  Mael,  Connerley  and 
Morath  (1996).  They  studied  biodata  inventory  content  in  a  job  selection  situation  to  see  what 
would  be  considered  invasive  from  a  privacy  standpoint.  They  found  that  personnel  selection 
professionals  and  potential  applicants  alike  considered  several  lines  of  questioning  to  be 
inappropriate.  (They  may  also  be  illegal,  see  Avoid  Illegal  Questions  below.)  Questions  about 
sexuality  or  intimacy,  such  as  age  of  first  sexual  activity,  frequency  of  dating  or  having  a 
miscarriage  were  considered  most  invasive,  followed  by  questions  about  specific  denominational 
religious  activities.  Also  considered  intrusive  were  questions  about  physical  or  mental  health 
such  as  suicide  attempts,  or  to  a  lesser  extent,  a  heart  attack  in  the  family,  partly  because  they 
may  trigger  the  reoccurrence  of  trauma.  In  an  expanded  second  study  with  students,  military 
personnel  and  subject  matter  experts,  the  investigators  found  similar  results.  Politics  was  an  area 
added  to  the  second  study.  Asking  about  membership  in  a  specific  party  was  considered  very 
invasive  while  having  been  a  political  volunteer  was  less  so,  but  still  rather  invasive.  Other 
topics  considered  invasive  by  many  second  study  participants  were  father’s  occupation,  being 
chronically  overweight  (but  not  by  military),  longest  stable  relationship,  and  relationship  with 
same  sex  parent.  Mael  et  al.  provide  the  following  suggestions:  1)  Avoid  asking  about  personal 
events  outside  the  workplace  or  other  public  arena,  especially  if  they  could  traumatize  or 
stigmatize  the  respondent.  2)  Replace  denomination-specific  questions  about  religion  with  more 
general  questions  about  spirituality.  3)  Replace  questions  about  political  affiliation  with  more 
general  questions  about  political  involvement.  4)  Avoid  questions  about  intimate  behavior  if 
they  are  likely  to  stigmatize  or  shame  the  respondent,  or  involve  contentious  topics. 


Avoid  Illegal  Questions:  Usually,  if  there  is  a  legitimate  need  to  include  very  intrusive 
content,  survey  researchers  can  do  so  on  a  limited  basis.  They  can  usually  elicit  accurate 
information  about  threatening  topics  due  to  their  pledge  to  protect  privacy  and  because  their 
intent  is  not  to  limit  opportunities  or  to  punish.  The  ability  to  include  intrusive  questions  is  also 
facilitated  by  the  anonymity  of  most  surveys  and  the  fact  that  survey  developers  usually  do  not 
know,  or  interact  directly  with,  respondents. 

Illegal  questions  are  a  different  story.  If  they  are  included,  respondents  have  every  right 
not  to  respond.  As  discussed  in  the  shadowbox  below,  illegal  questioning  is  discussed  in  the 
following  two  contexts:  1)  on  employment  applications  or  other  evaluative  surveys  where 
discrimination  may  result  and  opportunities  may  be  denied  based  on  responses  to  certain 
irrelevant  questions,  and  2)  on  any  survey,  concerning  job  irrelevant  organizational 
memberships,  where  First  Amendment  rights  to  free  speech  and  assembly  may  be  constrained 
and  where  harassment  of  respondents  (or  fear  of  it)  may  result  from  knowledge  of  such 
affiliations  by  potentially  hostile  parties. 


Employment  Application  Form  or  Other  Evaluative  Survey.  Sometimes  the  purpose  of  a 
survey  (or  “application  form”)  is  to  evaluate  individuals.  In  these  situations,  certain  questions  or 
items  are  illegal,  or  have  to  be  asked  only  in  certain  ways.  Laws  have  been  enacted  to  protect 
people  from  discrimination  and  other  untoward  outcomes  when  filling  out  evaluation 
applications  or  being  interviewed  prior  to  selection  for  a  job,  for  leasing  or  buying  property,  or 
for  obtaining  credit.  See  Medley  (1984)  for  a  list  of  these  laws  for  California  and  New  York. 
They  are  representative  of  similar  laws  in  other  states.  Campion  and  Arvey  (1989)  also  discuss 
taboo  behaviors  (subject  to  litigation)  in  an  interview  which  may  be  used  in  conjunction  with  the 


21 


application.  Smart  (1983)  discusses  legal  considerations  and  provides  questions  that  can  and 
cannot  be  asked. 

Alderman  and  Kennedy  (1995,  pp.  277-290;  see  also,  Meyers  (1992)  and  Nachman  & 
Ryan,  1994,  p.  25)  provide  an  account  of  one  of  the  first  serious  challenges  to  psychological 
testing  for  pre-employment  screening.  (Psychological  tests  frequently  are  a  form  of  survey.)  The 
case  (Soroka  v.  Dayton  Hudson  Corp.),  involved  a  person  applying  to  be  an  asset  protection 
specialist  (guard  to  prevent  shoplifting)  for  a  large  national  department  store.  The  individual 
was  asked  to  take  a  704-item  Psychscreen  inventory  which  contained  very  personal  items  (for 
instance,  about  sexual  matters  like  “I’ve  never  indulged  in  any  unusual  sex  practices”  and 
religious  beliefs  like  “  I  believe  in  the  second  coming  of  Christ”).  This  instrument  had  been 
formed  by  extracting  items  from  other  instruments  developed  decades  ago  (the  Minnesota 
Multiphasic  Personality  Inventory  or  MMPI  and  the  California  Personality  Inventory  or  CPI). 
Many  items,  especially  from  the  MMPI,  were  developed  to  assess  psychopathology  and  were 
intended  to  be  used  for  clinical  purposes.  Soroka  considered  many  items  ridiculous  and 
irrelevant  to  his  application  for  employment.  He  was  so  upset  he  photocopied  the  instrument  to 
provide  evidence  of  how  bad  the  items  were.  He  then  sued  the  store.  The  California  court  of 
appeals,  backed  by  a  strong  state  constitution  with  good  privacy  protection  provisions,  found  the 
store  did  not  demonstrate  a  compelling  interest  or  a  job-related  purpose  (the  nexus  requirement) 
in  asking  applicants  to  respond  to  these  kinds  of  items.  The  store  settled  out  of  court,  and  while 
not  admitting  liability,  it  stopped  using  the  test  for  this  type  of  applicant  in  California  and  created 
a  $  1 ,300,000  fund  to  compensate  the  plaintiff  and  other  Californians  who  had  taken  Psychscreen. 

As  Alderman  and  Kennedy  (1995)  indicate,  the  psychological  exam  used  was  argued  to 
be  a  medical  exam.  Soroka  and  his  legal  team  believed  the  Americans  With  Disabilities  Act 
would  support  this  view,  providing  additional  grounds  for  restricting  the  use  of  such  instruments 
to  prevent  employment  discrimination  against  disabled  persons. 

Examples  of  items/questions  or  behaviors  that  may  be  illegal  or  constrained  (depending 
on  jurisdiction)  on  an  evaluation  application  or  in  an  evaluation  interview  are: 


age 

ancestry 
arrest  record 
birth  control  usage 
birthplace 

citizenship  in  a  specific  country  other  than  United  States  (e.g.,  items  about  naturalization 
papers  or  your  parents’  citizenship) 
disability  not  related  to  work 
ethnic  group 

family,  spouse,  children,  relatives  (e.g.  items  about  specific  information  about  family 

members  or  relatives  who  are  not  employees  of  the  company) 

gender 

language  (e.g.,  native  language  or  how  a  foreign  language  was  learned) 
marital  status 

mental  illness  or  treatment  by  a  mental  health  professional 
military  experience  (outside  the  United  States) 

miscellaneous  (anything  not  related  to  work  or  employment  eligibility) 


•  name  (original  or  maiden) 

•  national  origin 

•  organizational  membership  (not  considered  relevant  by  the  applicant  to  job  performance  or 
memberships  related  to  race,  religion,  national  origin,  etc.) 

•  pregnancy;  plans  to  have  children 

•  physical  limitations  not  relevant  to  work 

•  race 

•  religion  or  religious  views 

•  sexual  orientation,  views  or  behaviors 

•  request  for  sexual  favors 

It  is  also  sometimes  unlawful  to  ask  questions  that  would  indirectly  reveal  these  facts.  In  some 
instances,  sensitive  questions  are  unlawful  if  asked  directly.  For  example,  “What  is  your  age”  is 
prohibited.  However,  it  is  all  right  to  inquire  more  broadly,  for  instance,  if  a  person  is  “between 
18  and  65  years  of  age.”  Likewise,  you  cannot  ask  about  an  arrest  record,  but  you  can  ask  about 
convictions  if  there  is  legitimate  “business  necessity”  involved.  However  a  conviction  is  usually 
not  grounds  for  being  denied  employment,  unless  directly  tied  to  the  job  or  if  it  would  put  an 
employer  at  considerable  risk  (Medley,  1984;  Smart,  1983). 

Organizational  Membership.  As  indicated  above,  asking  for  information  about  job 
irrelevant  organizational  memberships  may  be  illegal  on  evaluation  forms  such  as  job 
applications.  Such  questioning  may  also  be  unlawful  on  constitutional  grounds. 

Bond  (1994)  wrote  about  two  cases  demonstrating  the  legal  precedent  prohibiting  forced 
disclosure  of  membership  in  private  groups.  These  cases  involved  the  First  Amendment  rights  of 
free  speech  and  assembly.  They  also  pertain  to  privacy  rights  (see  Shattuck,  1977,  Chapter  2, 
for  a  discussion  of  the  privacy  of  association  and  belief  and  several  other  examples).  Both  cases 
involved  groups  protecting  their  membership  lists  from  disclosure  to  government  entities.  The 
first  case  involved  the  National  Association  for  the  Advancement  of  Colored  People  (NAACP) 
versus  the  State  of  Alabama  in  1958.  The  second  case  occurred  in  1994  and  involved  the  Texas 
Knights  of  the  Ku  Klux  Klan  versus  the  Texas  Commission  on  Human  Rights.  For  further 
discussion  of  the  First  Amendment  and  privacy  rights  rationale  for  not  asking  for  job-irrelevant 
organizational  membership,  see  Appendix  E. 

Organizational  membership  lists  could  be  amassed  from  survey  data.  Given  the  legal 
precedents  mentioned  and  possible  harm  to  people  due  to  intolerance  of  the  associations  and 
beliefs  of  others  (see  Appendix  E),  I  recommend  that  requests  for  specific  information  about  job- 
irrelevant  organizational  affiliations  be  avoided. 

Ethical  Implications  for  Prospective  Survey  Participants 

Ethical  implications  have  been  discussed  from  the  perspective  of  the  obligations  of 
survey  developers  to  participants.  Focusing  now  on  survey  participants,  I  contend  they  have  the 
right  to  decline  to  participate,  completely,  or  in  response  to  specific  items/questions,  if  they 
believe  any  of  the  following  conditions  are  true: 

•  The  survey  is  not  voluntary:  respondents  are  told  they  must  respond  and  they  desire  not  to, 
or  other  forms  of  coercion  are  used,  such  as  the  threat  of  punishment. 
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•  Insufficient  information  is  given  upon  which  participants  could  come  to  a  decision  based  on 
informed  consent. 

•  The  survey  is  requested  as  a  part  of  one’s  job,  but  some  or  all  content  is  not  job  relevant, 
including  questions  about  job  irrelevant  organizational  memberships  or  privileged 
information  the  respondent  desires  not  to  share. 

•  Privacy  may  not  be  protected:  the  survey  is  not  anonymous,  confidentiality  may  be 
breached,  analysis  and  reporting  of  non-narrative  data  may  occur  at  the  level  of  the 
individual  respondent  or  at  a  low  level  of  aggregation  which  might  permit  disclosure  of 
identity,  or  personal  identifying  information  may  not  be  excised  from  narrative  data,  except 
where  the  identity  of  a  prominent  individual  (e.g.,  a  CEO  or  Commander)  would  be  difficult 
to  conceal. 

•  The  requested  information  may  be  used  to  discriminate  unfairly  against  respondents,  harass 
them  or  harm  them  in  other  ways. 

•  The  items  or  questions  are  needlessly  intrusive,  or  they  are  too  offensive. 
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Chapter  5 


Selecting  a  Sample 

And  Arranging  for  Distribution  and  Return 
Random  Sampling  Versus  a  Census 

Sampling  is  a  complex,  technical  issue  that  can  lead  to  misleading  findings  if  not  done 
well.  Sampling  is  briefly  covered  here.  For  more  detail,  consult  Sudman  (1983).  The 
discussions  of  sampling  by  Bradbum  and  Sudman  (1988)  and  Salant  and  Dillman  (1994)  are  also 
recommended.  They  are  practical  and  easy-to-read. 

Surveying  only  to  a  portion  of  the  population  is  called  sampling.  If  you  survey  the 
entire  population  of  interest,  you  have  a  census.  Unless  your  population  is  small,  you  will  likely 
need  a  sample  rather  than  a  census.  The  incremental  statistical  benefit  of  gaining  more  subjects 
falls  off  as  your  sample  gets  large.  Much  more  so  than  interviews,  surveys  are  well  suited  to 
sampling  large  populations 

Sometimes  a  census  is  taken,  even  with  large  populations,  for  political  rather  than 
statistical  reasons.  People  like  to  be  consulted  on  important  issues  which  impact  them.  If  they 
are  left  out,  they  may  be  annoyed.  Thus,  sometimes  the  decision  is  made  to  go  to  an  entire 
population  so  everyone  can  express  themselves.  With  survey  automation  this  is  more  and  more 
feasible.  For  instance,  the  AF  Surgeon  General  decided  that  the  OMG  Survey  should  go  to 
nearly  everyone  in  the  Air  Force  Medical  Service.  This  included  personnel  working  outside 
medical  facilities  since  implementation  of  the  OMG  structure  was  a  hot  issue  that  most  people  in 
the  medical  service  were  expected  to  have  strong  opinions  about.  If  you  contemplate  surveying 
an  entire  population,  consider  whether  it  is  worth  the  cost.  Also  ask  yourself  if  the  issue  is  one 
about  which  most  people  would  want  to  provide  input.  People  lament  being  “surveyed  to  death.” 
Thus,  also  consider  if  your  target  population  was  surveyed  about  other  issues  in  the  recent  past. 
Adjust  your  schedule,  or,  if  necessary,  your  sampling  strategy. 

The  representativeness  of  a  sample  is  critical.  Representativeness  means  that  your 
sample  adequately  reflects  or  represents  the  target  population  of  interest.  Several  means  are 
used  to  achieve  a  representative  sample.  For  instance,  you  may  make  sure  your  survey  is  random 
so  that  everyone  has  an  equal  chance  of  inclusion.  You  may  first  stratify  your  sample  and  then 
select  randomly  within  the  strata.  Sometimes  a  random  sample  may  not  be  appropriate.  You 
may  want  your  sample  to  be  more  targeted.  For  instance,  maybe  you  know  there  is  little 
variance  in  your  sample  on  a  particular  measure.  An  example  is  performance  ratings,  a  common 
criterion  variable  despite  often  being  inflated.  To  correct  for  this  problem,  you  might 
intentionally  select  people  whose  scores  differ  on  this  performance  measure.  As  discussed 
below,  you  may  also  choose  purposeful  sampling  rather  than  random  sampling. 

Sometimes,  representativeness  is  violated.  For  instance,  professors  sometimes  have 
limited  access  to  funds  and  to  diverse  groups  of  people  to  serve  as  subjects.  However,  they  have 
a  captive  population  of  students  who  tend  to  be  young,  Caucasian,  and  middle  to  upper  class.  So 
professors  tend  to  use  these  subjects  extensively  in  their  research  and  probably  get  results  of 
limited  generalizability.  Also,  the  responses  provided  may  be  distorted  due  to  the  status 
imbalance  between  students  and  professors  and  the  fact  that  the  students  are  known  to  the 
investigator. 
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As  another  example,  the  Literary  Digest  incorrectly  predicted  a  Landon  victory  over 
Roosevelt  for  president  in  1936  (Fishkin,  1995;  Salant  &  Dillman,  1994;  Dillman,  1978).  This 
prediction  was  based  on  a  huge,  unscientific  sample  of  people  who  had  either  telephones  or  cars. 
In  those  days  only  about  35%  of  the  population  had  phones.  Cars  were  also  less  common  than 
today.  People  with  these  items  tended  to  be  affluent.  Less  affluent  people,  who  were  more 
likely  to  vote  for  Roosevelt,  were  left  out  of  the  sample.  Fishkin  (1995)  provides  a  detailed 
account  of  this  debacle.  Interestingly,  the  magazine  based  its  tabulation  on  millions  of  postcards 
solicited  from  the  biased  sample  described.  During  this  period,  George  Gallup  gained  national 
recognition  by  challenging  the  Literary  Digest.  He  publicly  predicted  the  magazine  would  be 
way  off  the  mark  in  their  estimate.  Promising  to  give  his  newspaper  clients  their  money  back  if 
he  failed,  he  demonstrated  he  could  predict  the  election  outcome  much  more  accurately  using  a 
scientifically  selected  sample  of  only  a  few  thousand  people.  Gallup  accurately  predicted 
Roosevelt’s  victory  and  the  success  of  his  firm  is  history.  The  Literary  Digest  went  out  of 
business. 

In  a  recent  telephone  example  (U.S.  News  and  World  Report.  1995),  people  were 
reported  to  use  their  answering  machines  to  screen  pollsters,  just  like  they  would  undesirable 
callers.  This  could  make  obtaining  a  representative  sample  via  phone  difficult. 

Another  example  where  representativeness  is  routinely  ignored  is  in  the  popular  use  of 
self-selected  listener  opinion  polls  (see  Fiskin  1995,  pp.  38-39).  Here,  radio  or  television 
audiences  are  asked  to  call  in  to  express  their  agreement  or  disagreement  with  a  question 
presented  to  them,  or  to  provide  their  reaction  to  an  event.  In  such  unscientific  polls,  participant 
reactions  would  likely  be  unrepresentative  of  those  of  the  general  population. 

As  mentioned  above,  a  way  to  obtain  a  representative  sample  is  to  generate  a  random 
sample.  In  a  random  sample  each  element  (usually  a  person)  in  the  population  is  independent 
and  everyone  has  an  equal  chance  of  being  selected.  Having  a  random  sample  would,  for 
instance,  ensure  that  everyone  exposed  to  an  intervention  had  an  equal  chance  to  provide  you 
with  feedback  on  it,  rather  than  the  survey  going  only  to  those  most  likely  to  favor  it.  Random 
sampling  is  appropriate  when  the  investigator  wishes  to  generalize  from  a  sample  to  a 
population.  If  you  want  to  ensure  that  different  parts  of  the  population  are  well  represented  you 
can  use  stratified  sampling  where  the  population  is  divided  into  parts  and  the  sample  is  drawn 
randomly  from  each  of  these  “stratum.”  See  also  the  discussion  of  purposeful  sampling  below. 

Your  response  rate  may  be  low.  When  high  quality  paper-and-pencil  surveys  are  used,  it 
has  been  the  author’s  experience  that  only  about  50%  to  70%  of  the  sample  respond.  Thus,  you 
should  survey  at  least  twice  as  many  people  than  needed  for  analysis  purposes.  Some  subgroups 
may  be  especially  underrepresented  due  to  their  small  size  in  the  population  and  response  rate 
problems.  One  solution  is  intentional  oversampling  of  selected  groups  such  as  minorities  and 
women.  Although  the  situation  is  improving,  women  have  historically  been  underrepresented  in 
positions  of  senior  leadership  (Kanter,  1977).  Therefore,  if  you  wish  to  examine  women  in 
senior  positions,  you  may  need  to  select  all  women  in  such  positions. 

The  goal  of  representativeness  was  described  above  in  terms  of  selecting  a  representative 
sample  at  a  single  point  in  time.  As  Fishkin  (1995,  p.  4)  reminds  us,  if  you  want  to  accurately 
represent  opinions,  a  single  snapshot  in  time  may  not  provide  the  best  view.  Although  some 
constructs  are  likely  to  be  rather  consistent  over  time  (for  example,  interests,  values  and 
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personality),  others  may  be  rather  transitory  (for  example,  satisfaction  with  supervision  or 
perceptions  of  other  organizational  climate  dimensions).  Moreover,  some  constructs  (for 
example,  opinions  about  political  office  holders  or  candidates)  may  fluctuate  widely  over  a  short 
time.  Thus,  to  achieve  a  representative  view  and  monitor  trends,  you  may  want  to  gather  data 
from  a  representative  sample  on  multiple  occasions.  Surveying  in  this  manner,  using  the  same 
or  different  subjects  each  time,  is  called  longitudinal  research.  Such  research  can  be 
accomplished  with  interviews  or  surveys,  but  is  especially  easy  when  automated  techniques  are 
used.  Sometimes  longitudinal  data  gathering  takes  place  over  a  brief  time,  say,  a  few  days 
before  an  after  a  special  event  or  an  election.  At  other  times  the  interval  is  several  months  to 
several  years,  for  example  when  measuring  retention  intent  or  quality  progress. 


Purposeful  Sampling 

Patton  (1980,  pp.  100-106)  contrasts  a  variety  of  forms  of  purposeful  sampling,  which  are 
intentionally  not  random.  This  form  of  sampling  is  appropriate  when  the  investigator’s  purpose 
is  to  understand  select  cases  without  generalizing  to  all  cases.  Here  “case”  may  be  an 
intervention  or  program  at  a  particular  site.  A  program  may  have  been  instituted  nationwide. 
You  may  know  it  is  working  well  at  most  locations.  However,  at  a  few  sites  it  is  disastrous. 
Rather  than  choosing  a  random  sample,  you  could  select  a  purposeful  sample  and  go  to  selected 
sites  for  intensive  study. 

There  are  multiple  forms  of  purposeful  sampling.  Patton  describes  the  option  of 
studying  extreme  cases— one  or  more  of  the  best  and  the  worst  of  them.  They  do  not  even  have 
to  be  randomly  selected  from  the  best  and  worst  sites.  They  can  be  the  ones  you  are  likely  to 
learn  the  most  from.  Another  approach  he  describes  is  studying  the  typical  case,  especially  if 
limited  resources  prevent  getting  information  from  enough  sites  to  make  generalizations. 

Instead,  you  get  in-depth  information  from  a  few  typical  programs.  Another  strategy  he 
describes  is  maximum  variation  sampling,  referring  to  individuals  within  a  site,  or  sites  within 
a  region.  For  example,  programs  could  be  spread  throughout  a  state  in  urban,  suburban  and  rural 
locations.  The  interviewer  may  not  have  enough  funds  to  gather  a  truly  random,  representative 
sample  of  sites.  However,  he  or  she  could  increase  the  variation  or  diversity  in  the  sample. 
Likewise,  people  at  a  site  who  have  been  identified  by  staff  to  have  had  quite  different 
experiences  with  an  intervention  could  be  selected  to  be  interviewed  or  surveyed.  This  would 
allow  the  investigator  to  better  understand  variations  in  the  program  and  the  experiences  of 
different  people  affected  by  it. 


Size 


Sample  size  is  an  important  consideration,  as  is  the  diversity  of  the  sample.  With  regard 
to  diversity  (see  the  discussion  of  representativeness  on  25),  care  should  be  taken  to  ensure  that 
the  full  variability  of  attitudes,  beliefs  or  behaviors  of  interest  is  represented  in  the  sample, 
usually  by  selecting  a  wide  array  of  different  types  of  people.  This  goal  is  more  easily  achieved 
if  the  sample  is  large.  Regarding  size,  survey  researchers  are  usually  able,  for  practical  reasons, 
to  have  access  to  more  people  than  interviews.  If  you  are  only  going  to  perform  total-sample 
analyses  and  not  divide  your  sample  into  subgroups  for  further  analyses,  you  can  get  by  with 
fewer  subjects.  Likewise,  fewer  subjects  would  be  acceptable  if  you  only  wish  to  compute 
simple  descriptive  statistics  without  performing  inferential  analyses. 
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At  the  Armstrong  Laboratory,  scientists  often  want  to  perform  complex  inferential 
analyses  on  entire  samples  as  well  as  subgroups.  They  usually  have  the  advantage  of  being  able 
to  obtain  large  samples.  Thus,  some  of  the  author’s  colleagues  are  uncomfortable  with  fewer 
than  200-300  people  in  each  analysis  subgroup.  Most  investigators  do  not  have  this  opportunity. 
Sudman  (1983)  recommends  at  least  100  cases  in  major  subgroups  and  at  least  20-50  cases  in 
smaller  subunits.  In  consulting  applications,  the  author’s  experience  is  that  you  take  steps  to 
oversample  small  groups.  If  subgroups  are  much  too  small  (say  fewer  than  ten  cases)  do  not 
report  these  results  if  doing  so  would  pose  confidentiality  problems.  However,  if  your  small 
group  comprises  the  entire  population  of  a  subunit  being  studied,  you  would  be  confident  about 
your  findings  since  you  would  no  longer  be  dealing  with  a  sample  and  you  would  not  be  making 
inferences.  But  there  is  another  problem:  as  you  perform  more  multiway  analyses,  your  chances 
of  having  cells  with  extremely  small  (thus,  person-identifiable)  frequencies  increases 
dramatically.  In  a  recent  survey  analysis,  my  colleagues  and  I  decided  not  to  share  output  of  a 
four- way  distribution  (organization  by  job  group  by  grade  range  by  item  response)  with  our 
client  because  individuals  could  have  been  identified  due  to  small  frequencies  in  some  cells. 

Salant  and  Dillman  (1994,  p.  55)  provide  a  useful  table  for  determining  the  sample  size 
to  use  at  different  levels  of  sampling  error  and  population  size.  For  example.  If  you  required 
only  +  or  -  3%  error  at  a  95%  confidence  level,  a  sample  size  of  516  is  recommended  if  your 
population  is  1,000  and  you  use  a  rather  conservative  estimate  of  variability  (50/50  split). 
Assuming  the  same  variability,  a  sample  size  of  964  would  be  needed  if  your  population  is 
10,000.  If  your  population  is  25,000  to  100,000,000  you  need  a  sample  of  between  1,023  and 
1,067  people.  For  very  large  populations  about  the  same  sample  size  would  be  needed  regardless 
of  population  size.  Keep  in  mind  that  the  probable  (but  often  unknown)  variance  in  the 
population  on  the  characteristics  being  measured  is  an  important  consideration  in  determining 
sample  size.  The  larger  the  variance  the  greater  the  need  for  a  large  sample.  Be  aware,  as  was 
recently  pointed  out  in  an  article  critical  of  polls  (telephone  surveys),  if  your  sample  is  not  truly 
random,  your  desired  margin  of  error  for  full  sample  or  subsample  analyses  may  not  be  satisfied 
(U.S.  News  and  World  Report.  1995). 

Mailout  Guidance  for  Paper-and-Pencil  Surveys 

Once  you  have  identified  your  sample,  arrange  for  the  distribution  and  return  of  your 
surveys.  You  may  only  have  to  distribute  a  small  number  of  surveys  to  people  employed  locally 
at  a  particular  facility,  returning  them  through  your  base’s  or  your  company’s  mail  distribution 
system.  However,  the  process  is  usually  more  cumbersome.  You  may  have  to  mail  surveys 
individually  to  organizational  addresses  throughout  the  world,  or  in  bulk  to  points  of  contact 
(POCs).  These  POCs  then  distribute  surveys  to  respondents  at  their  work  sites  or  serve  as  group 
survey  administrators.  Sometimes  POCs  also  motivate  participants  and  collect  and  return 
surveys  (or  scan  sheets  and  comments  pages).  Sometimes  they  even  check  to  see  if  sheets  are 
ready  for  scanning.  However,  the  practice  of  having  survey  materials  examined  or  returned 
through  POCs  could  be  a  threat  to  confidentiality.  You  may  also  not  want  POCs  to  see  the 
content  of  comments  pages  before  you  do,  since  you  have  not  had  a  chance  to  excise  personal 
identifying  information.  If  you  do  not  have  POCs,  or  you  do  not  want  them  to  return  survey 
materials,  have  participants  send  their  responses  directly  back  to  you,  typically  through  the  mail. 
You  should  include  an  address  labeled  return  envelope.  Even  if  you  have  POCs  collect  and 
return  responses  and  take  steps  to  safeguard  confidentiality,  allow  respondents  to  return  their 
materials  directly  to  you.  They  can  do  this  if  they  remain  fearful  about  the  possible  lack  of 
confidentiality. 
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If  you  use  a  survey  POC,  consider  having  the  POC  arrange  for  group  administration  at  a 
central  location.  This  approach  usually  increases  response  rate.  However,  some  respondents 
may  consider  group  administration  a  means  of  forcing  participation.  Let  those  who  wish  to  leave 
do  so.  Encourage  your  POC  to  select  a  good  location  and  specify  the  desired  characteristics  of 
the  administration  site.  Since  survey  developers  often  have  limited  control  over  where  surveys 
are  administered,  locations  may  be  cramped,  not  private,  too  noisy,  too  hot  or  cold,  etc. 
Sometimes  people  are  handed  a  survey  or  scan  sheet  with  nothing  suitable  to  write  on.  For 
example,  auditoriums  without  tables  or  even  pop-up  desks  are  sometimes  used.  This  is  not 
recommended.  A  conference  room,  a  school  room,  a  club  facility  or  a  cafeteria  with  good 
tables  and  comfortable  surroundings  is  better.  Guidance  for  group  survey  administrators  is 
provided  in  Appendix  F. 

If  you  request  the  assistance  of  POCs,  keep  in  mind  this  may  be  an  extra  duty  that  may 
displease  them.  Try  to  secure  their  cooperation  ahead  of  time  and  work  with  them  to  minimize 
the  burden  your  survey  places  upon  them  and  their  staffs.  If  you  provide  POCs  with  specific 
guidance  on  who  should  receive  the  surveys  (rather  than  letting  them  identify  subjects)  give 
them  sufficient  information  to  easily  identify  the  persons  and  their  organizational  locations. 
Names  and  organizational  symbols  may  not  be  sufficient. 

Direct  mailing  of  surveys  via  regular  mail  to  each  individual  respondent  at  their 
organizational  address  is  often  recommended.  This  allows  the  investigator  to  maintain  more 
control  over  who  the  specific  recipients  of  the  survey  will  be  without  having  to  rely  on  others  to 
follow  the  sampling  plan,  or  to  select  for  themselves  who  the  recipients  will  be.  Respondents 
could  complete  the  survey  on  company  time  and  return  it  directly  to  the  survey  developer 
without  an  intermediary  who  may  threaten  their  sense  of  confidentiality.  You  would,  however, 
have  to  be  careful  about  getting  accurate  organizational  addresses  and  go  through  the  labor- 
intensive,  tedious  process  of  stuffing  envelopes  with  surveys  and  self-addressed  (preferably 
stamped  or  postage  free)  return  envelopes.  You  may  also  have  to  include  a  separate  cover  letter 
and  optical  scan  sheet  if  they  are  not  a  part  of  your  survey.  Upon  return,  you  would  begin 
another  tedious  process  of  opening  envelopes,  and  removing  surveys,  comments  pages  and 
answer  sheets. 


Guidance  for  Automated  and  Telephone  Surveys 

When  using  automated  surveys,  make  sure  you  have  accurate  e-mail  addresses  if  you 
distribute  your  survey  that  way.  You  could  also  ship  disks  to  POCs  around  the  world  and  have 
them  distribute  disks  to  the  sample  (or  census).  Respondents  could  either  send  back  the  disks 
directly  in  a  labeled  mailer  you  provide,  or  return  them  to  the  POCs.  If  POCs  are  part  of  the 
return  loop,  take  steps  to  safeguard  confidentiality.  For  instance,  self-generated  passwords  could 
be  included  with  encryption  of  responses.  Sometimes  multiple  respondents  may  be  able  to 
provide  their  responses  on  a  single  disk.  This  may  represent  a  threat  to  confidentiality  and 
veracity  since  computer  literate  colleagues  may  be  able  to  find  and  change  the  responses  of 
others. 

After  preparing  your  automated  survey,  make  sure  it  works  on  everyone’s  local  or  wide 
area  network,  not  just  on  yours,  if  you  distribute  it  that  way.  If  you  use  disks,  make  sure  they  are 
compatible  with  participants’  equipment,  and  make  sure  respondents  know  how  to  access  and 
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exit  the  software  and  return  data  they  provide.  Make  sure  that  any  branching  capability  works 
for  the  subpopulations  for  which  it  was  intended. 

Disks  will  likely  need  to  be  duplicated  in  large  numbers  and  this  task  can  be  contracted. 
The  contractor  can  label  the  duplicated  disks  with  the  survey  title,  organizational  identification 
and  brief  instructions  on  how  to  enter  the  survey.  Additional  instructions  can  be  included  within 
the  survey  itself.  The  contractor  can  also  package  and  ship  the  disks  for  you,  with  instructions  to 
POCs  and  paper  copies  of  the  cover  letter  for  each  respondent.  They  can  also  include  labeled 
return  mailers  to  protect  the  disks  should  respondents  be  more  comfortable  sending  the  disks  to 
you  directly  rather  than  returning  them  to  the  POC  on  site.  The  contractor  can  create  survey 
packets  which  neatly  contain  the  survey  and  cover  letter  inserted  in  the  return  mailer.  This 
makes  the  task  of  distributing  the  materials  easier.  Of  course,  you  can  also  contract  for  data  tape 
creation  and  analyses  if  you  wish.  A  brief  Statement  of  Work  for  disk  duplication  and 
distribution  is  provided  as  Appendix  G.  An  example  of  general  instructions  for  POCs  is  included 
as  Appendix  H. 

If  you  are  using  an  automated  survey,  viruses  can  be  a  problem.  This  is  not  as  critical  if 
only  one  person  records  responses  on  a  single  disk.  However,  all  electronic  files  should  be 
checked  for  viruses  before  they  are  distributed,  at  the  administration  site  and  after  they  are 
returned.  The  potential  virus  problem  can  be  greater  if  several  people  provide  responses  on  the 
same  disk  because  the  virus  could  wipe  out  the  responses  of  these  participants  at  once.  The  more 
computer  systems  involved,  the  greater  the  chance  of  being  infected  with  a  virus.  Viruses  are 
less  of  a  problem  on  networks  because  they  usually  include  software  to  check  for  them.  If  you 
contract  to  have  your  disks  duplicated,  virus  checking  is  typically  part  of  the  service.  In  a  recent 
field  test  of  the  Armstrong  Laboratory’s  automated  survey  technology,  approximately  one 
percent  of  returned  disks  had  viruses  and  about  half  of  these  were  from  a  single  site. 

Companies  with  a  computer-assisted  telephone  interviewing  (CATI)  capability  usually 
have  elaborate  ways  for  identifying  respondent  phone  numbers  and  using  the  cheapest  available 
carrier  to  call  them.  They  will  sometimes  alert  people  ahead  of  time  with  a  post  card  and  will 
even  provide  a  1-800  number  for  respondents  to  call  back  at  a  convenient  time. 

With  automated  surveys,  you  can  track  who  did  and  did  not  respond.  This  enhances 
your  ability  to  follow  up  with  nonrespondents  and  to  assess  response  bias. 

Timing 

The  timing  of  your  survey  is  an  issue  to  consider.  You  may  wish  to  survey  only  one 
time,  or  you  may  need  multiple  administrations  to  perform  longitudinal  analyses.  With  multiple 
administrations,  you  can  detect  change  and  monitor  trends  over  time.  Note  Fishkin’s  (1995) 
comment  earlier  (page  26)  about  doing  a  better  job  of  representing  opinions  if  you  take  multiple 
“snapshots”  in  time.  Consider  if  the  attitudes  or  other  attributes  you  wish  to  measure  are  likely 
to  be  stable  over  time.  If  this  is  likely,  you  may  gain  little  from  multiple  administrations.  If  they 
are  likely  to  be  unstable,  multiple  “waves”  of  survey  administration  and  analysis  would  be 
beneficial.  Facets  of  personality,  core  values,  and  political  ideology  tend  to  be  relatively  stable. 
General  attitudes,  say  about  work,  can  be  enduring  or  fluctuating  depending  on  the  facet  focused 
upon.  For  instance,  you  may  be  quite  satisfied  with  your  work  over  a  long  period,  but  your 
attitude  toward  your  supervisor  may  fluctuate  based  on  his  or  her  treatment  of  and  support  for 
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you.  Attitudes  toward  specific  political  figures  and  about  world  events  tend  to  fluctuate.  For 
better  or  worse,  it  is  little  wonder  why  politicians  have  become  so  concerned  about  their 
“approval  ratings”  in  the  polls. 

If  you  perform  a  program  evaluation  or  intervene  to  improve  a  situation  based  on  initial 
survey  results,  you  will  want  to  collect  data  before  and  after  to  determine  if  desired  changes  in 
attitudes,  behaviors  or  other  factors  took  place. 

To  prevent  surveying  your  target  population  too  much,  consider  if  it  just  received 
another  survey.  Also,  consider  if  responding  to  the  other  survey  may  influence  responses  to  your 
survey  and  space  yours  accordingly.  If  you  need  multiple  survey  administrations,  spread  them 
out  if  you  use  a  paper-and-pencil  format.  For  instance,  monitor  quality  progress  once  or  twice  a 
year  at  the  most.  If  you  survey  multiple  times  over  short  intervals— to  study  reaction  to  fast¬ 
changing  events— use  automated  procedures. 

Be  aware  that  when  a  survey  is  administered  may  influence  the  results.  People  are  often 
surveyed  after  a  critical  event  (say  a  catastrophe).  Especially  if  it  was  a  senseless,  cruel  or 
avoidable  catastrophe,  like  an  act  of  terrorism,  attitudes  toward  the  event  may  be  particularly 
strong  and  inflexible  at  first  and  the  intensity  of  emotions  may  be  temporarily  elevated. 
Therefore,  to  get  a  more  accurate  assessment  of  attitudes  and  emotions  related  to  the  event,  and 
their  changes  over  time,  it  is  wise  to  also  measure  these  constructs  at  multiple  points  in  time- 
weeks,  months  and  even  years  later. 
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Chapter  6 


Some  Initial  Decisions  on  Survey  Format 
Select  Your  Type  of  Survey 

You  need  to  decide  upon  the  type  of  survey  you  wish  to  use.  Do  you  want  a  paper-and- 
pencil  survey  or  an  automated  survey?  If  you  selected  an  automated  approach,  will  it  involve  a 
skilled  interviewer  as  in  Computer  Assisted  Telephone  Interviewing  (CATI)?  If  you  select  a 
telephone  survey,  will  a  paper-and-pencil  survey  administered  over  the  phone  suffice,  or  do  you 
need  CATI?  See  Chapter  2  for  guidance  to  aid  your  decision. 

Decide  if  You  Need  Optical  Scanning 

If  you  use  a  paper-and-pencil  survey,  decide  if  respondents  are  to  provide  responses  on  a 
separate  or  embedded  form  which  can  be  scanned,  or  if  they  can  just  write  or  mark  their 
responses  on  a  nonscannable  survey.  Sometimes  separate  scan  sheets  are  used.  Sometimes  the 
survey  booklets  themselves  are  scannable.  If  a  paper-and-pencil  format  is  used,  surveys  with  an 
embedded  scanning  capability  are  preferred  to  ones  with  separate  optical  scan  sheets.  It  is 
cumbersome  for  respondents  to  constantly  shift  back  and  forth  between  the  survey  and  an  optical 
scan  sheet,  having  to  find,  each  time,  the  appropriate  item  number  and  response  oval.  But  having 
the  scanning  built  in  may  be  costly  or  time  consuming.  If  the  number  of  items  and  sample  size 
are  small,  marking  in  a  nonscannable  survey  may  be  acceptable.  This  would  require  later 
transcription  of  responses  to  a  data  sheet  or  electronic  file.  If  you  use  a  separate  op-scan  sheet 
and  have  a  comments  section,  have  respondents  tear  out  the  comments  page  and  return  it  with 
the  op-scan  sheet.  Also,  if  you  use  a  separate  general  purpose  scan  sheet,  make  sure  your  items 
don’t  have  more  response  options  than  there  are  ovals  (or  whatever)  on  the  scan  sheet  for  each 
particular  item.  The  number  of  marking  places  varies  from  place  to  place  on  some  commercially 
available  general-purpose  scan  sheets  and  this  simple  concern  can  be  overlooked.  Also,  make 
sure  that  the  scan  sheet  you  use  can  be  scanned  on  your  machine  or  one  available  to  you.  If  you 
allow  multiple  responses  to  some  items,  make  sure  your  scanner  can  handle  this. 


If  you  use  automated  survey  technology  or  Computer- Assisted  Telephone  Interviewing 
(CATI)  you  can  avoid  the  need  for  optical  scanning  or  transcribing  of  your  data  by  creating  an 
electronic  data  base  as  you  survey.  In  CATI,  you  can  even  do  real-time  data  veracity  checks  as 
the  person  is  being  surveyed. 


Reduce  Reading  Difficulty  and  Eliminate  Language  Barriers 

Early  in  the  survey  development  process  consider  the  range  of  reading  ability  in  your 
target  population  and  tailor  your  survey  to  the  lowest  reading  level.  In  many  instances,  you 
cannot  expect  your  target  population  to  read  as  well  as  you  do.  You  don’t  need  to  give  reading 
tests.  If  you  are  developing  an  organizational  survey,  talk  to  managers  or  personnel  specialists 
about  reading-level  requirements  for  employment  and  about  the  general  workforce  literacy.  For 
surveys  in  other  settings,  talk  to  knowledgeable  people  (such  as  community  leaders)  about  the 
literacy  and  educational  attainment  of  those  to  be  surveyed.  Do  so  in  a  way  that  is  not 
condescending. 
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Reading  ability  may  not  be  your  only  concern.  Do  you  have  people  in  your  target 
population  for  whom  English  is  a  second  language?  Can  they  perhaps  speak  only  a  little  English 
and  hardly  read  it  at  all?  If  so,  you  will  have  to  go  a  step  further  and  develop  one  or  more 
foreign  language  versions  of  your  survey  if  the  number  of  people  who  fall  into  this  category  is 
large.  If  the  number  is  small,  you  could  have  a  translator  assist  them  instead. 

If  you  translate  a  survey,  have  it  done  by  people  who  are  fluent  in  both  English  and  the 
language  into  which  the  survey  is  translated.  Have  other  knowledgeable  people  review  the 
translation  to  make  sure  wording,  grammar  and  spelling  are  accurate. 

If  you  make  a  foreign  language  survey  available  to  some  members  of  your  target 
population  or  if  you  provide  a  translator,  do  this  with  sensitivity  to  the  feelings  of  those  involved. 
Respondents  should  not  be  humiliated  or  embarrassed. 

Select  the  Background  Information  You  Need 

You  will  probably  want  demographic  information.  Demographics  are  background  data 
such  as  age,  income,  education,  etc.,  that  are  used  to  summarize  the  composition  of  a  population. 
This  may  include  name  and  social  security  number  (SSN).  As  discussed  on  page  19,  avoid 
asking  for  name  and  SSN,  if  possible.  Protect  this  information  from  disclosure  if  you  do  collect 
it.  Although  this  is  not  an  exhaustive  list,  other  commonly  requested  demographic  data  are: 

•  Age 

•  Base,  company,  continental  United  States  (CONUS)  or  overseas  location 

•  Education  (years,  highest  degree,  high  school  graduate,  GED,  or  certification) 

•  Ethnic  or  racial  group 

•  Family  members  (number) 

•  Grade  or  rank 

•  Gender  (a  preferred  term  to  sex) 

•  Income  (individual  or  household;  corporate  profit) 

•  Industry  (manufacturing,  telecommunications,  utilities,  etc.) 

•  Major  Command  (MAJCOM) 

•  Management  or  staff  position 

•  Marital  status 

•  Military  or  civilian  status;  enlisted  or  officer 

•  Occupation,  profession,  or  technical  specialty  (job  series  or  Air  Force  Specialty  Code) 

•  Position  (more  specific  than  just  management  versus  staff) 

•  Product  or  service  type 

•  Retention  intent  (willingness  to  stay  with  the  organization) 

•  Separation  or  retirement  date  (anticipated) 

•  Tenure,  usually  in  years  (federal.  Air  Force,  organizational,  career  field  job,  etc.) 

Some  demographic  information  may  be  considered  too  intrusive  or  objectionable  and 
may  be  withheld.  Respondents  may  object  to  indicating  their  race  or  ethnicity  having 
experienced  bigotry  from  others.  They  may  feel  their  marital  status  or  income  (see  Make 
Objectionable  Items  More  Acceptable  on  page  55)  is  none  of  your  business.  Asking  for  these 
data  may  be  restricted  by  law  in  situations  such  as  personnel  selection  (see  Avoid  Illegal 
Questions  on  page  21). 
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Sometimes  you  need  not  ask  for  demographic  information  since  it  can  be  obtained  from 
personnel  files  or  other  sources.  This  will  shorten  your  survey;  however,  make  sure  you  have 
the  identifying  information  necessary  to  retrieve  these  data. 

Demographic  data  are  usually  collected  for  the  purpose  of  performing  subgroup 
analyses.  The  attitudes,  values,  behaviors  and  experiences  of  people  in  different  demographic 
groups  usually  differ.  These  differences  would  be  obscured  if  only  total  group  analyses  were 
performed.  Demographic  data  also  function  as  predictor  (independent)  or  criterion  (dependent) 
variables  in  theoretical  models  which  survey  researchers  sometimes  test  empirically. 

Divide  Your  Survey  into  Parts 

Surveys  often  have  multiple  parts.  For  instance  background  (demographic)  information 
could  be  in  one  section  and  attitudes  toward  whatever  you  are  studying  could  be  in  another.  A 
comment  section  at  the  end  of  your  survey  is  recommended.  In  the  USAF  Medical  Service 
Objective  Medical  Group  (OMG)  Survey  the  following  parts  were  included: 


•  Part  1 . 

•  Part  2. 

•  Part  3. 

•  Part  4. 

•  Part  5. 

•  Part  6. 


Background  Information 

Views  About  OMG  and  Its  Implementation 

Views  About  the  Future  of  the  Group  Staff 

Personal  Satisfaction  Since  OMG  Implementation 

Views  About  the  Location  of  Specific  Functions  or  Services 

General  Comments 


On  the  OMG  Survey,  branching  was  used.  Branching  refers  to  automatically  going  to  a 
particular  question,  group  of  questions,  or  section  based  on  demographic  information  or  how  a 
person  responds  to  specific  attitudinal  or  behavioral  items.  Branching  is  easiest  to  accomplish 
on  automated  surveys  or  computer  assisted  telephone  interviews.  For  example,  in  the  automated 
OMG  Survey,  some  items  were  only  for  senior  leaders.  The  survey  automatically  presented 
these  items  to  only  senior  leaders  in  a  manner  that  was  transparent  to  all  respondents. 

You  could  do  something  similar  on  a  paper-and-pencil  survey  by  having  such  questions 
at  the  end  and  instructing  those  who  are  not  senior  leaders  to  go  on  to  the  comments  section. 
Clustering  items  for  a  specific  subgroup  was  employed  on  the  paper-and-pencil  San  Antonio 
Air  Logistics  Center  Career  Planning  Survey.  Here,  items  were  placed  together  at  the  end  of 
the  survey  which  only  retirement  eligible  people  were  asked  to  respond  to.  Others  were  asked  to 
go  on  to  the  comments  page.  Such  clusters  are  intentionally  placed  at  the  end  of  a  paper-and- 
pencil  survey,  especially  if  a  separate  optical  scan  sheet  is  used.  If  you  use  a  separate  scan  sheet, 
do  not  ask  respondents  to  skip  items;  they  could  lose  their  place. 

When  you  divide  your  survey  into  parts,  this  division  should  be  intentionally  general  in 
nature.  Dividing  surveys  into  sections  on  the  basis  of  the  constructs  or  themes  being  measured 
(for  instance,  clustering  all  items  measuring  a  single  construct  like  “communication  flow”  or 
“job  satisfaction”  together  and  then  labeling  these  small  clusters)  is  not  recommended.  You 
may  want  to  cluster  some  similar  items  together  for  contrast  effect  if  they  are  nearly  identical. 
But  you  need  not  place  all  items  measuring  similar  themes  together.  Instead,  intentionally 
intersperse  such  items  with  others  throughout  the  survey.  Why  is  this  approach  recommended? 
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As  much  as  possible,  survey  participants  should  respond  to  the  specific  item  before  them  rather 
than  to  some  general  impression  of  the  theme  it  may  be  partly  measuring.  If  like  items  (thought 
to  tap  a  single  construct)  are  together,  their  co-location  may  influence  how  a  person  responds  by 
conveying  an  implicit  idea  of  what  your  theme  may  be.  Placing  headings  in  the  survey 
identifying  each  construct  measured  heightens  this  problem  by  explicitly  identifying  each  theme. 

The  interspersing  of  items  of  similar  content  throughout  the  survey  also  provides  a 
veracity  check.  If  items  measuring  similar  themes  are  dispersed,  respondents  will  be  less  aware 
of  how  they  responded  to  similar  items  previously.  If  responses  to  these  dispersed  items  are 
consistent,  respondents  are  probably  taking  the  survey  seriously  and  responding  in  a  way  which 
reflects  how  they  really  think  or  feel. 

Often  items  are  clustered  on  the  basis  of  the  scale  used.  For  instance,  you  may  cluster 
your  agreement  scale  items  together  and  then  cluster  items  measuring  different  facets  of 
satisfaction  together  using  a  satisfaction  scale.  This  reduces  the  distraction  and  cognitive 
processing  needed  to  mentally  shift  frequently  from  one  scale  to  another,  making  the  survey 
easier  and  less  time  consuming  to  complete. 
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Chapter  7 


Item  Construction 


Item  Stems  and  Responses 

There  are  two  parts  of  an  item:  its  stem  and  its  response  options.  Some  items  use  no 
predetermined  response  options  at  all;  the  response  is  filled  in  by  the  respondent.  Some  items 
use  a  unique  set  of  response  options  designed  specifically  for  the  item  and  others  use  standard 
scales  common  to  several  items.  For  an  extensive  treatment  of  item  form,  wording  and  content, 
see  Schuman  and  Presser  (1981). 

For  ease  of  interpretation  and  analysis,  survey  developers  usually  provide  items 
composed  of  stems  (the  actual  item)  and  a  set  of  exhaustive  options  from  which  respondents 
select  the  best  or  most  appropriate  response.  These  are  called  “objective”  response  options 
because  they  are  not  subjectively  formulated  by  respondents.  Rather,  all  survey  participants  get 
to  choose  from  the  same  set  of  fixed  response  options  for  an  objective  comparison  across 
subjects.  This  has  the  advantage  of  providing  survey  developers  with  easy  aggregation  and 
statistical  manipulation.  They  can  perform  quantitative  rather  than  qualitative  analyses. 

Objective  items  also  have  disadvantages.  They  force  participants  to  respond  in  words, 
phrases,  and  categories  selected  by  the  survey  developer.  These  words  or  categories  may  not  fit 
the  respondent’s  situation  exactly.  Thus,  distortion  may  be  introduced.  Another  form  of 
response  is  the  open-ended  narrative.  It  is  a  subjective  form  of  responding  which  can  be  time 
consuming  and  difficult  to  analyze.  But  it  allows  people  to  respond  in  their  own  words. 

Items  with  Narrative  Responses 

It  is  not  always  possible,  convenient,  or  even  desirable  to  get  the  information  you  need 
solely  from  objective  items.  Short  narrative  items  are  often  included.  Sometimes  respondents 
provide  an  objective  response  to  an  item  and  a  narrative  comment  about  it.  Narrative  responses 
or  comments  are  “subjective”  because  they  are  in  the  respondent’s  own  words.  Here,  the 
respondents  may  complete  a  sentence.  Alternatively,  they  may  fill  in  letters  or  numbers  to 
designate,  for  example,  grade,  rank  or  job  specialty  code.  They  may  also  provide  a  verbal 
statement  ranging  from  a  word  or  phrase  to  several  paragraphs.  If  you  request  narrative 
responses,  keep  most  short  (a  phrase  to  a  few  sentences)  by  limiting  writing  space  since  they  are 
time  consuming  to  provide  and  difficult  to  analyze.  Also  allow  respondents  to  provide  longer 
narratives,  for  instance  in  a  final  comments  page  or  at  the  end  of  selected  sections  of  your 
survey.  For  general  comments,  provide  a  page  of  unobstructed  space.  I  have  received  general 
comments  ranging  from  a  few  words  to  carefully  crafted  letters  several  pages  in  length. 
Sometimes  these  comments  pages  are  signed  with  an  accompanying  phone  number  or  e-mail 
address,  and  the  survey  developer  or  client  who  commissioned  the  survey  is  requested  to  contact 
the  respondent  to  discuss  issues  further. 

Items  with  Item-Specific  Response  Options 

The  response  options  which  the  survey  developer  provides  are  objective  response 
options.  These  are  in  contrast  to  subjective  narrative  responses  provided  by  respondents,  as 
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discussed  above.  Some  objective  response  options  are  common  across  items;  others,  discussed 
here,  are  specific  to  an  item. 

Before  item-specific  response  options  are  discussed,  let’s  clarify  what  a  stem  is.  An 
item  stem  is  the  actual  statement  or  question  to  which  the  respondent  is  asked  to  select  or 
provide  a  response.  For  instance: 

I  have  to  rely  on  the  grapevine  (rumors)  to  get  information  on  important  work-related  matters, 
or,  of  an  entirely  different  type. 

When  stationed  at  Los  Angeles  Air  Force  Base,  which  of  the  following  services  provided  by  the 
Family  Support  Center  have  you  used?  (Select  all  services  you  have  used.) 

are  both  item  stems.  The  first  uses  a  standard  agreement  scale  (discussed  in  the  next  section), 
whereas  the  second  uses  the  item-specific  response  options  below: 

A.  Information  and  referral 

B.  Spouse  employment 

C.  Relocation  support 

D.  Air  Force  Aid  (financial  assistance) 

E.  Family  Life  Education 

F.  Family  Services  (loan  of  household  goods) 

G.  Single  Parent  Support  Group 

H.  Not  applicable,  I  have  not  used  these  Family  Support  Center  services 

Note  that  a  “Not  applicable”  option  is  included  to  make  the  options  exhaustive.  It  is  listed  last 
because  respondents  should  typically  read  the  options  before  coming  to  a  not  applicable 
conclusion  about  use  of  services.  However,  one  could  argue  that  this  option  should  be  placed 
first  if  the  survey  developer  was  confident  respondents  would  know  immediately  whether  they 
did  or  did  not  use  such  facilities.  Putting  the  escape  option  first  reduces  response  time  and 
cognitive  processing  demands.  Such  a  response,  called  an  “escape  option,”  is  discussed  more 
fully  below. 

Another  example  of  a  stem  with  item-specific  response  options  is: 

If  you  have  been  planning  to  leave  the  Air  Force,  how  seriously  did  you  look  for  another  full¬ 
time  job  during  the  past  year? 

A.  Not  applicable;  I  plan  to  stay  in  the  Air  Force. 

B.  I  did  not  look  at  all 

C.  I  looked,  but  not  seriously 

D.  I  looked  seriously 

E.  I  looked  very  seriously 

Note  that  in  this  example,  the  “Not  applicable”  option  is  put  first  since  respondents  would  not 
have  to  review  other  options  before  knowing  their  inapplicability.  Respondents  would  know 
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immediately  if  they  intended  to  stay  in  the  Air  Force.  Note  also  that  for  brevity,  response 
options  were  truncated  by  excluding  the  phrase  “for  another  job”  in  each  option.  The  phrase  “for 
another  full-time  job”  in  the  stem  suffices. 

Items  with  Common  Response  Scales 

I  have  a  preference  for  Likert  scales  (Likert,  1 932),  named  after  Rensis  Likert,  a  survey 
pioneer.  This  type  of  scale  is  discussed  below.  Likert  scales  are  popular,  having  been  used 
extensively  for  decades.  They  are  also  easy  to  construct.  There  are  other  types  of  scales,  such  as 
Guttman  scales  and  Thurstone  scales.  For  a  discussion  of  these  and  other  types  of  scales,  see 
DeVellis  (1991),  Anderson,  Basilevsky  and  Hum  (1983),  Edwards,  (1983),  or  Chapter  15  of 
Nunnally  (1978).  For  a  detailed  historical  presentation  of  scaling  techniques,  see  Maranell 
(1974).  Guttman  and  Thurstone  scales  are  difficult  to  construct  and  not  the  method  of  choice  for 
the  measurement  of  attitudes  (Nunnally,  1978).  Although  once  widely  used  in  psychology  and 
sociology,  especially  in  the  1930’s  and  1940s,  they  are  rarely  used  today  (Anderson,  Basilevsky 
and  Hum,  1983;  Nunnally,  1978).  They  have  limited  applicability  and  their  disadvantages 
appear  to  outweigh  their  advantages  (DeVellis,  1991).  Thus,  they  are  not  discussed  further  here. 
The  semantic  differential  (Snider  &  Osgood,  1969;  Nunnally,  1978)  is  another  approach  to 
scaling  which  is  still  frequently  used  and  easy  to  construct.  It  is  briefly  discussed  here  as  an 
alternative  format. 

You  may  sometimes  want  to  use  multiple  scales  with  the  same  set  of  item  stems.  The 
importance  versus  satisfaction  distinction  is  frequently  studied  since  satisfaction  with  facets  of  a 
work  environment  or  product  that  are  not  very  important  are  less  critical  to  deal  with  than  facets 
which  are  perceived  to  be  very  important.  Thus,  you  may  want  to  ask  how  satisfied  respondents 
are  with  facets  of  whatever  is  being  investigated  and  then  ask  how  important  these  facets  are  to 
them.  You  may  also  want  to  contrast  how  things  were  in  the  past  with  how  they  are  now,  or  how 
they  are  now  with  how  they  should  be  or  are  anticipated  to  be  in  the  future.  In  these  instances, 
you  would  use  the  same  or  very  similar  items  modified,  if  necessary,  to  conform  to  the  wording 
of  the  different  scales.  The  common  items  could  be  presented  with  the  multiple  scales 
simultaneously  or  sequentially.  Sequential  presentation  is  usually  preferred  since  respondents 
are  less  easily  influenced  by  their  response  to  the  previous  scale.  However,  presenting  multiple 
scales  simultaneously  reduces  the  length  of  the  survey.  Also,  you  may  want  respondents  to  make 
a  conscious  comparison  and  simultaneously  assess  satisfaction  and  importance  of  various  facets 
of  the  workplace.  Alternatively,  respondents  could  be  asked  to  simultaneously  compare  how 
their  work  environment  was  in  the  past  with  how  it  is  in  the  present  and  how  they  anticipate  it 
will  be  in  the  future.  A  marketing  firm  may  want  potential  customers  to  contrast  the  features 
they  have  on  their  present  car  with  those  they  would  like  to  have  on  a  new  car.  Judicious  use  of 
multiple  scales  with  the  same  or  similar  items  is  acceptable,  but  don’t  overdo  it.  Such  scales  add 
length  and  complexity  and  may  reduce  the  willingness  of  respondents  to  complete  your  survey. 

A  Recommended  Likert  Scale  Format;  Sometimes  you  will  need  to  write  items  with 
their  own  unique  set  of  response  options.  However,  it  is  recommended  that  you  use  a  common 
scale  for  multiple  items  as  much  as  possible.  Using  a  common  scale  makes  taking  a  survey 
easier  and  faster  and  requires  less  arduous  cognitive  processing.  Consider  using  the  popular 
Likert  format.  Likert  scales  are  simple  to  use  and  “verbally  anchor”  each  scale  point.  This 
means  that  a  verbal  statement  is  provided  for  each  response  choice,  such  as  “Agree”  or  “Strongly 
Agree,”  allowing  the  researcher  to  report  specifically  the  meaning  of  each  response  selected. 
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When  a  paper-and-pencil  survey  is  used,  I  recommend  portraying  a  Likert  scale 
graphically  along  a  response  scale  continuum  if  the  scale  is  to  be  used  for  responses  to  several 
items.  An  example  of  such  a  scale,  with  lead-in  statement,  is  provided  below. 


Example  of  the  Author’s  Recommended  Likert  Scale 


Use  the  scale  below  to  indicate  your  agreement  or  disagreement  with  each  of  the  following 
statements.  If  you  don’t  know,  or  a  statement  is  not  applicable  to  you,  mark  response  “G.” 


A 

B 

C 

D 

E 

F 

G 

Strongly 

Disagree 

Slightly 

Slightly 

Agree 

Strongly 

Don’t  Know  or 

Disagree 

Disagree 

Agree 

Agree 

Not  Applicable 

This  example  uses  a  common  agreement  scale.  Satisfaction  scales  are  also  quite  common. 
Examples  of  other  Likert  scales  are  provided  in  Appendix  G.  On  paper-and-pencil  surveys,  the 
response  scale  should  always  be  in  sight.  Therefore,  repeat  it,  with  lead-in,  at  the  top  of  a  new 
page.  If  a  respondent  sees  two  pages  at  once  and  the  scale  is  clearly  visible  on  the  page  to  the 
left,  it  need  not  be  repeated  on  the  right.  Do  not  overdo  the  number  of  different  scales  you  use 
on  a  survey.  Also,  do  not  expect  respondents  to  constantly  shift  back  and  forth  among  scales. 
This  would  be  confusing  and  place  an  unnecessary  cognitive  processing  burden  on  them.  Select 
one  or  a  few  scales  and  cluster  items  using  a  particular  scale  together. 

On  a  paper-and-pencil  survey,  respondents  typically  fill  in  or  bubble  in  a  response 
corresponding  to  the  letter  or  number  associated  with  the  verbally  anchored  response  point  they 
select.  I  have  shown  A  through  G  letters  because  they  are  common  on  optical  scan  sheets.  But 
numbers  can  also  be  used.  If  a  researcher  desires  only  to  tabulate  frequencies  and  percents  of 
people  responding  to  different  response  options,  there  would  be  no  need  to  convert  letters  to 
numbers.  But  usually  descriptive  or  inferential  statistics  are  desired  which  involve  numerical 
calculations.  In  such  instances  the  letters  are  converted  to  numbers  so  responses  can  be 
analyzed.  Typically  what  would  be  considered  by  whomever  commissioned  your  survey  to  be 
the  most  desirable  response  is  assigned  the  highest  number.  Thus  in  the  scale  shown  above, 
typically  A=1,B  =  2,  C  =  3,D  =  4,  E  =  5  and  F  =  6,  except  in  cases  where  reverse  scoring  is 
appropriate  such  as  when  more  of  something  (like  strong  agreement)  is  not  desirable  from  the 
perspective  of  your  customer.  This  would  be  the  case  for  negatively  worded  items  and  is 
discussed  more  fully  in  the  section  on  Quantitative  Descriptive  Analyses  and  Results  Reporting, 
page  70. 


Automated  Survey  Presentation  of  a  Likert  Scale.  On  automated  surveys,  items  are 
usually  presented  alone  on  the  screen,  rather  than  clustered  with  other  items  on  a  page. 
Therefore,  the  Likert  scale  is  typically  presented  with  each  item  stem  as  discrete  response 
options.  The  presentation  is  not  usually  graphic.  That  is,  a  line  is  not  used  to  represent  the 
response  scale  continuum  like  in  the  example  presented  above.  Also,  an  alphabetic  or  numeric 
designation  is  not  needed  for  each  response.  Automated  survey  response  options  can  be  placed 
vertically  or  horizontally.  Some  software  packages  allow  you  to  start  the  cursor  at  any  point 
along  the  scale.  It  can  even  go  to  a  particular  point  at  or  near  the  midpoint  of  the  scale  with 
arrows  pointing  in  each  direction  along  with  a  message  to  select  a  response.  This  would  lessen 


39 


the  possible  concern  that  always  starting  at  one  end  of  the  scale  may  bias  responding  by 
encouraging  respondents  to  choose  options  in  that  part  of  the  scale.  For  example,  some  people 
have  contended  that  starting  with  the  most  negative  response  option  leads  to  responses  that  are 
more  negative  (see  Response  Scale  Directionality  below). 


Inclusion  of  an  Escape  Option:  Notice  that  in  the  example  on  page  39  a  “Don’t  Know 
or  Not  Applicable”  option  is  included  as  response  “G.”  This  option,  or  something  like  it,  is 
called  an  “escape  option”  and  is  almost  always  included.  When  such  a  response  is  present  in  a 
graphic  depiction  of  a  Likert  scale,  I  recommend  that  the  response  scale  continuum  not  be 
extended  to  this  response.  This  is  because  this  escape  option  response  is  not  part  of  the  scale. 
Sometimes  (for  instance,  with  satisfaction  items)  I  use  just  the  “Not  Applicable”  statement  since 
people  should  know  their  personal  satisfaction  with  facets  of  the  workplace.  Let’s  consider  one 
of  the  author’s  “lessons  learned.”  On  the  Physician  Leadership  Survey  develop)ed  for  the  AF 
Surgeon  General,  I  intentionally  left  off  an  escape  option  on  very  general  items  concerning 
physician  leadership  using  an  agreement  scale  and  on  items  using  importance  and  willingness 
scales,  reasoning  that  the  escape  option  didn’t  make  sense  in  these  situations.  To  my  surprise, 
physician  respondents  continued  to  select  the  escape  option  by  marking  “G”  on  their  answer 
sheet,  even  though  it  did  not  exist  for  these  items  on  the  survey.  Apparently,  they  had  gotten 
used  to  it  on  earlier  items,  wanted  it,  and  continued  to  use  it. 

In  some  situations,  a  “No  Opinion”  escape  option  may  be  appropriate,  especially  with 
statements  of  political  attitudes.  Instead  of  including  a  “No  Opinion”  response  option,  an 
alternative  approach  would  be  to  tell  respondents  in  a  lead-in  statement  to  respond  only  to  items 
about  which  they  have  an  opinion.  This  was  done  on  several  measures  presented  by  Robinson, 
Rusk  and  Head  (1973).  A  potential  drawback  is  they  may  not  read  the  lead-in  or  they  may  lose 
their  place  if  using  a  separate  optical  scan  sheet.  Having  participants  only  respond  to  items  about 
which  they  have  an  opinion  may  be  a  way  to  avoid  the  question  of  whether  or  not  to  use  a  neutral 
midpoint  (see  page  42). 

The  escape  option  is  handled  for  analysis  and  results  reporting  in  different  ways. 
Sometimes  those  selecting  the  escape  option  are  not  included  in  the  analysis  of  that  particular 
item.  Sometimes  the  percent  of  persons  selecting  this  option  are  reported  to  management  on 
histogram  charts  along  with  those  selecting  other  options.  Quite  often  those  selecting  “Don’t 
Know  or  Not  Applicable”  are  recoded  as  having  selected  the  midpoint  of  the  scale  (3.5  in  this 
case)  and  are  included  for  analysis  purposes.  If  this  approach  is  chosen,  respondents  are  treated 
as  if  they  gave  a  “Neither  Agree  Nor  Disagree”  response,  even  though  the  format  shown  above 
avoids  a  neutral  midpoint.  But  the  “Don’t  Know  or  Not  Applicable”  response  is  not  a  neutral 
response  and  is  certainly  not  positioned  that  way  on  the  scale.  This  approach  is  mentioned  only 
because  it  is  common.  If  there  were  many  such  cases,  it  would  tend  to  draw  more  valid 
responses  toward  the  midpoint  and  is,  therefore,  not  recommended.  The  issue  of  how  to  deal 
with  escape  options  for  analysis  purposes  needs  further  empirical  study. 


Response  Scale  Directionality:  It  is  common  for  Likert  scales  to  start  with  the  negative 
or  with  a  zero  point  on  the  left  (or  first)  and  go  to  the  positive  or  more  of  something  on  the  right 
(or  last).  Recently,  a  manager  in  the  field  who  reviewed  the  author’s  paper-and-pencil  draft  of 
the  USAF  Medical  Service  Objective  Medical  Group  (OMG)  Survey  objected  to  this.  He 
contended  that  to  start  with  the  negative  or  with  a  little  of  something  may  bias  the  results  in  a 
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negative  direction  and  suggested  that  most  surveys  start  with  agreement  and  go  to  disagreement. 
Others  on  the  OMG  survey  team  shared  this  concern. 

This  concern  seemed  reasonable  so  I  discussed  the  matter  with  colleagues  and  examined 
the  literature.  Starting  with  the  positive  and  going  to  the  negative  is  not  the  case  in  the  Air  Force 
and  seldom  the  case  in  academic  surveys.  The  most  frequently  used  format  is  the  one 
recommended  here.  The  approach  used  by  different  investigators  varied;  some  switch  the  scale 
around,  but  most  do  not.  Rossi,  Wright  and  Anderson  (1983)  show  a  Likert  scale  starting  with 
“Strongly  disagree”  on  the  left  (scored  as  1)  going  to  “Strongly  agree”  on  the  right,  scored  as  5 
(or  more  for  more  scale  points),  unless  a  negative  item  would  call  for  reverse  scoring.  Also,  as 
advocated  here,  each  point  is  verbally  anchored.  Nunnally  ( 1 978)  provides  examples  of 
agreement  scales  with  disagreement  on  the  left  or  first  in  a  sequence  and  agreement  to  the  right 
or  last. 

After  talking  with  colleagues  and  examining  the  literature,  I  remained  confident  that  my 
approach  was  sound.  However,  this  is  a  research  issue  that  could  be  examined  empirically. 
Development,  administration  and  analysis  of  the  OMG  Survey  was  on  too  quick  a  time  table  to 
include  examination  of  scale  directionality  in  the  research  design.  The  survey  was  also 
automated  in  its  final  form  so  that  initially  the  cursor  placed  the  respondent  in  the  middle  of  the 
scale  for  each  item,  making  the  directionally  issue  mute.  However,  in  future  paper-and-pencil  or 
automated  survey  research,  surveys  could  be  developed  in  two  formats  with  verbal  anchors  going 
from  “Strongly  Disagree”  to  “Strongly  Agree”  and  vice  versa.  The  survey  could  be  administered 
to  two  samples  matched  on  demographic  characteristics.  Differences  in  average  response 
patterns  and  response  variance  could  be  examined  to  see  if  starting  with  negative  verba!  anchors 
on  the  left  of  the  response  scale  results  in  more  negative  responses. 


Scale  Points  and  the  Neutral  Midpoint;  The  number  of  scale  points  used  on  a  Likert 
scale  varies  and  is  usually  five  or  six.  The  author’s  colleague.  Dr.  R.  Bruce  Gould,  performed 
some  research  several  years  ago  which  indicated  that  more  scale  points  were  advantageous  up  to 
a  maximum  of  about  11.  Nunnally  (1978)  reports  similar  findings.  He  states  that  reliability 
increases  with  the  number  of  scale  points,  rapidly  at  first,  leveling  off  at  about  7  scale  points, 
with  little  gain  after  1 1  points.  Another  colleague  with  experience  in  occupational  surveys,  Mr. 
William  J.  Phalen,  commented  recently  that  the  optimal  number  of  points  may  depend  on  the 
attributes  of  the  respondent  and  on  the  object  or  issue  being  evaluated.  For  instance,  it  may 
depend  on  how  knowledgeable  respondents  are  about  what  is  being  rated  and  how  easy  or  hard  it 
is  for  them  to  make  fine  distinctions  about  what  is  being  evaluated.  Some  things  can  be  easily 
measured  with  only  a  few  categories;  others  require  many. 

Some  survey  experts  recommend  use  of  a  neutral  midpoint.  Others  do  not.  Nunnally 
(1978)  explains  that  a  neutral  midpoint  (such  as  “Neither  Agree  Nor  Disagree”)  may  make 
respondents  more  comfortable  because  they  may  have  a  neutral  reaction  to  your  statement. 
However,  he  also  argues  that  use  of  a  neutral  option  may  introduce  response  styles  unrelated  to 
the  attitude  being  measured.  He  reports  that  in  studies  in  which  respondents  were  given  a  5- 
point  scale  with  a  neutral  midpoint  at  first,  and  who  were  later  administered  a  6-point  scale  with 
no  neutral  midpoint,  there  was  reliable  variance  in  the  responses  of  subjects  who  first  chose  the 
neutral  midpoint.  This  would  suggest  that  not  having  a  midpoint  is  preferred.  However,  he 
concludes  this  is  not  a  very  important  issue  and  should  be  left  to  the  individual  investigator. 
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Presser  and  Schuman  (1980)  studied  the  effects  of  omitting  and  offering  middle 
positions  such  as  “About  the  same  as  now”  and  “Right  amount.”  While  these  options  differ  from 
the  neutral  midpoint  “Neither  Agree  nor  Disagree”  common  on  some  agreement  scales,  this 
study  sheds  light  on  the  effects  of  using  middle  positions  on  attitude  surveys.  Presser  and 
Schuman  accepted  middle  position  responses  on  “omitted”  interview  protocols  only  if  such 
responses  were  spontaneously  offered  by  the  respondent.  For  other  people  interviewed,  they 
explicitly  included  such  options.  The  midpoint  statements  were  sometimes  volunteered  under 
the  “omitted”  condition.  However,  when  they  were  explicitly  offered,  the  number  of  respondents 
selecting  such  a  category  typically  increased  by  10  to  20  percent  and  sometimes  up  to  39  percent. 
The  selection  of  “Don’t  know”  responses  declined  only  slightly  when  such  middle  statements 
were  provided.  The  greatest  impact  was  a  proportional  decline  in  the  selection  of  the  polar 
positions.  They  concluded  more  research  was  needed  to  study  form  effects  such  as  the  inclusion 
or  exclusion  of  middle  position  statements. 

I  once  used  5-point  scales  with  a  neutral  midpoint,  but  now  uses  6-point  scales  with  no 
neutral  midpoint,  preferrin  this  approach  for  reasons  discussed  in  the  shadowbox  below.  But  I 
agree  with  those  cited  above  that  this  remains  a  research  issue  for  future  investigation.  Also,  it 
could  be  argued,  as  my  colleague  Lt  Tim  Porter  does,  that  leaving  off  a  neutral  midpoint  forces 
people  with  no  opinion  on  an  issue  to  select  an  untruthful  response  of  mild  agreement  or 
disagreement.  If  you  wish  to  use  scales  with  a  neutral  midpoint,  examples  are  provided  in 
Appendix  G.  Another  approach  would  be  to  use  a  scale  with  no  neutral  midpoint,  but  to  add  a 
“no  opinion”  escape  option  in  addition  to  the  “don’t  know  or  not  applicable”  response. 

A  policy  concerning  Likert  scale  format  was  established  in  the  Air  Force  Materiel 
Command  (AFMC).  The  former  commander  of  AFMC  decreed  a  few  years  ago  that  the  author’s 
recommended  format  (6-point  with  no  neutral  midpoint)  would  be  the  standard  throughout  the 
command.  He  did  this  because  I  recommended  the  format  and  because  he  was  interested  in 
consistency  and  ease  of  comparing  results  across  surveys. 


Why  I  Recommends  Not  Using  a  Neutral  Midpoint?  In  1990,  the  leadership  of  what  was 
then  Space  Systems  Division  at  Los  Angeles  Air  Force  Base  asked  me  to  do  an  assessment  of  the 
quality  of  life  in  and  around  Los  Angeles.  Five-point  agreement,  satisfaction  and  importance 
scales  with  neutral  midpoints  were  used  (see  Appendix  G  for  examples).  When  results  were 
presented  to  management,  they  didn’t  know  how  to  interpret  the  “Neither  Agree  Nor  Disagree” 
response  for  their  purpose  of  trying  to  improve  the  quality  of  life  for  base  personnel.  They 
thought  it  was  a  wishy-washy,  useless,  cop-out  response.  They  even  asked  me  to  exclude  from 
analysis  people  who  had  selected  the  neutral  midpoint. 

The  viewpoint  of  these  managers  toward  the  neutral  midpoint  made  sense.  The  author’s 
scale  was  encouraging  people  not  to  take  a  stand  and  it  was  confusing  to  managers.  When  it 
comes  to  attitudes  about  issues  that  affect  people  personally,  they  usually  have  an  opinion. 
Therefore,  they  should  be  encouraged  to  take  a  stand  by  leaving  out  the  neutral  midpoint.  Of 
course,  they  should  always  be  able  to  select  the  “Don’t  Know  or  Not  Applicable”  option  if  they 
really  don’t  know  or  if  an  item  does  not  apply  to  them.  This  option  could  also  be  selected  if  they 
felt  they  had  no  opinion.  Therefore,  in  subsequent  surveys,  I  transitioned  to  the  6-point  format 
with  an  escape  option  but  no  neutral  midpoint.  Although  I  have  decided  to  use  such  a  scale  in 
preference  to  one  with  a  neutral  midpoint,  scales  with  neutral  midpoints  may  be  appropriate  on 
some  surveys.  For  instance,  they  may  be  better  suited  for  academic  surveys  concerned  with 
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model  testing  and  development  than  for  surveys  designed  to  provide  feedback  to  managers  or 
other  clients. 


Verbal  Anchors:  The  Likert  format  typically  uses  verbal  anchors  at  each  scale  point. 
Some  survey  scholars  prefer  to  verbally  anchor  only  the  ends  of  the  response  scale.  Others  may 
verbally  anchor  some  but  not  all  of  the  scale  points.  A  colleague,  Mr.  William  Phalen,  cautioned 
in  a  recent  conversation  that  respondents  may  tend  to  select  points  that  are  anchored  in 
preference  to  those  which  are  not.  This,  in  effect,  reduces  the  number  of  scale  points.  He  also 
cautioned  that  verbal  anchors  may  be  misunderstood  or  may  be  interpreted  differently  from 
anchors  of  similar  scales.  This  is  another  area  where  additional  research  is  needed. 

An  Alternative  Format:  The  Semantic  Differential:  I  have  typically  used  a  Likert 
scale,  as  discussed  above.  Another  popular  format  you  may  wish  to  consider  is  the  semantic 
differential.  Like  the  Likert  scale,  this  is  a  generic  term  for  a  scale  type  rather  than  for  a  specific 
scale.  The  scale  theme  or  verbal  anchors  vary,  but  the  format  remains  about  the  same.  This  type 
of  scale  has  multiple  points  (for  instance,  the  scale  shown  below  has  seven,  but  it  could  vary) 
which  are  anchored  only  on  the  extremes  by  bipolar  adjectives.  As  Nunnally  (1978)  points  out, 
the  semantic  differential  is  very  flexible  since  virtually  any  concept  that  can  be  named  can  be 
rated.  It  is  also  a  scale  form  which  is  easy  to  construct  and  quick  to  take.  An  example  of  a 
semantic  differential  scale  is  provided  below. 


An  Example  of  the  Semantic  Differential 
Members  of  The  United  States  Congress 


1 

Ineffective 

2 

3 

4 

5 

6 

7 

Effective 

Foolish 

Wise 

Dishonest 

. 

_  Honest 

Weak 

. 

_  Strong 

Useless 

. 

_  Useful 

Harmful 

. 

. 

. 

Helpful 

Anticipated  Future  Scale  Developments:  With  the  availability  and  sophistication  of 
computers  and  the  more  precise  measurement  they  afford,  it  is  likely  that  better  scaling 
techniques  will  be  developed  in  the  future.  For  instance,  computers  make  it  possible  for 
respondents  to  locate  responses  almost  anywhere  along  a  response  continuum;  not  just  at  one  of 
five  or  six  choice  points.  This  capability  is  not  widely  used.  The  response  scale  format 
recommended  in  this  text  has  the  drawback  that  respondents  are  limited  to  a  small  number  of 
verbally  described  choice  points  and  cannot  select  a  point  anywhere  along  the  continuum. 
Computers  can  provide  many  choice  points.  They  are  actually  discrete  points,  but  they  can  be 
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presented  in  a  way  that  they  appear  continuous  or  almost  continuous.  Perhaps  respondents  could 
touch  the  computer  screen  with  a  point  or  probe  at  any  desired  location  along  a  response 
continuum.  Regardless  of  the  number  of  verbal  anchors  used,  or  where  they  are  placed  along  the 
continuum,  the  ability  of  computer  technology  to  approximate  a  continuous  response  scale  may 
represent  a  step  forward  in  survey  technology.  Of  course,  this  will  need  to  be  examined 
empirically. 

What  about  other  scale  formats?  The  author’s  colleagues  have  experimented  with  other 
formats.  For  instance,  Drs.  Patrick  Kyllonen  and  Joshua  Herwitz  have  used  a  semicircular 
format  in  which  respondents  start  at  a  point  equidistant  from  each  response  option.  This  is  done 
to  reduce  response  bias.  Dr.  Winston  Bennett  recommends  a  similar  “speedometer”  approach  as 
well  as  a  “drag  a  line”  procedure.  Albert,  Phalen,  Selander,  Dittmar,  Tucker,  Hand, 
Weissmueller  &  Rouse  (1994)  experimented  with  new  scales  for  measuring  time  spent  on  tasks 
in  occupational  surveys.  These  scales  included  an  Absolute  Time  Spent  Scale,  a  End  Anchored 
Graphical  Scale,  a  Three  Stage  Scale,  a  Direct  Magnitude  Scale  and  a  Indirect  Magnitude  Scale. 


A  Special  Case:  Surveys  for  Studying  Emotion:  Arvey,  Renz,  Watson  and  Driskill 
(1997)  recommend  that  emotional  expression  in  organizations  be  studied  to  improve  the 
prediction  and  understanding  of  job  performance  and  other  outcomes.  Surveys  and  interviews 
are  two  methods  they  recommend  for  this  purpose.  They  describe  and  provide  excerpts  from  a 
number  of  surveys  used  to  measure  the  recall  of  emotions  and  emotional  episodes.  These 
measures  sometimes  use  Likert  scales  but  frequently  use  other  response  formats.  Although  these 
instruments  show  promise,  refinement  of  existing  measures  and  development  and  evaluation  of 
new  ones  appears  to  be  needed. 

Surveys  need  not  be  limited  to  investigating  the  recall  of  past  emotions.  They  can  also 
be  used  to  tap  attitudes  toward  and  reactions  to  the  emotional  displays  by  oneself  or  others. 

Also,  interviews  provide  another  good  method  for  studying  emotion. 


Item  Writing  Suggestions 

Crafting  item  ideas  into  good  items  is  challenging.  Guidelines  for  doing  this  are  provided 
below.  See  also  Dillman  (1978,  Chapter  3),  Schuman  and  Presser  (1981),  Frey  (1983,  Chapter 
4),  Sheatsley,  (1983),  Bradbum  and  Sudman  (1988,  Chapter  7)  and  Salant  and  Dillman  (1994, 
Chapter  6). 

Keep  Items  Short  and  Simple:  When  an  item  idea  is  first  written,  it  is  usually  too 
complex  and  too  long.  Take  your  initial  item  idea  and  identify  words  that  are  jargon  or  which 
may  not  be  easily  understood  by  all  respondents.  Remember  that  not  all  respondents  will  be  as 
educated  as  you  and  their  vocabulary  may  not  be  as  complete.  Your  survey  should  be  written  at 
a  reading  level  all  respondents  will  be  comfortable  with.  English  may  be  a  second  language  for 
some  respondents  and  you  may  need  to  develop  a  foreign  language  version  (see  Reduce  Reading 
Difficulty  and  Eliminate  Language  Barriers  on  page  32).  Substitute  simple  words  with  the  same 
meaning  for  complex  words.  If  an  acronym  is  used,  spell  it  out  first.  Next,  look  for  verbosity 
and  simplify  the  item  stem  (and  response  options  if  they  are  item-specific)  by  eliminating 
unnecessary  words  or  phrases.  For  example,  this  wordy  item: 
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We  are  naturally  experiencing  some  “growing  pains”  with  OMG  implementation,  but  the 
structure  is  basically  sound  and  we  will  work  out  our  problems. 


became  this  item: 


We  are  experiencing  growing  pains  with  OMG  implementation,  but  the  structure  is  basically 
sound. 


Another  way  to  simplify  an  item  is  to  reduce  redundancy.  Sometimes  there  is  a  common 
phrase  in  item-specific  response  options.  This  phrase  can  be  put  in  the  stem.  Likewise,  you  can 
take  a  phrase  that  would  be  common  to  many  items  and  extract  it  from  the  stem  as  a  separate 
lead-in.  For  instance  in  the  OMG  Survey,  we  had  more  than  fifty  items  with  the  common  lead- 
in  phrase: 


With  implementation  of  the  OMG  structure: 


When  items  are  used  with  a  common  lead-in,  do  not  capitalize  the  continuation  of  the  stem,  since 
it  simply  completes  the  sentence,  as  follows: 


With  implementation  of  the  OMG  structure: 

teamwork  among  medical  staff  members  improved. 


The  OMG  Survey  also  used  the  following  lead-in  phrase  for  satisfaction  items,  making 
the  continuation  of  the  stems  short: 


Since  OMG  implementation,  how  satisfied  have  you  been  with: 


Make  Items  Exhaustive  and  Mutually  Exclusive:  Objective  response  options  should 
be  exhaustive  (covering  the  complete  range  of  options)  and  mutually  exclusive  (not 
overlapping). 

Assume  on  a  paper-and-pencil  survey  that  you  want  to  ask  how  long  a  respondent  has 
been  on-the-job  (that  is,  his  or  her  tenure).  Let’s  say  you  are  only  interested  in  years  (rather  than 
years  and  months).  Listing  every  year  option  (say  1  to  20  years  or  more)  as  a  nearly  continuous 
variable  without  categorizing  age  into  multiple-year  groups  would  take  up  too  much  space.  It 
may  also  not  be  feasible  given  the  limited  number  of  options  on  most  scan  sheets.  So  you  would 
aggregate  years  into  ranges,  creating  a  more  categorical  variable.  To  be  exhaustive,  you  would 
include  a  “Less  than  one  year”  option.  To  be  mutually  exclusive,  you  would  avoid  saying  “Two 
to  four  years”  and  then  “Four  to  six  years,”  etc.,  because  four  years  would  be  included  in  two 
response  options.  Rather,  you  might  say  “Two  to  three  years,”  “Four  to  six  years,”  etc.  and  end 
with  “More  than  twenty  years.”  There  is  nothing  magical  about  twenty  years.  You  could 
continue  with  the  more  precise  categories  and  say  “More  than  thirty  years.”  Of  course,  you  may 
be  creating  too  many  options  and  could  correct  this  by  broadening  categories  (“Two  to  five 
years”  etc.).  Use  what  makes  sense  and  is  practical. 
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One  way  to  make  sure  the  alternatives  are  exhaustive  is  to  include  an  “Other”  option. 
On  an  automated  survey  you  could  say  something  like  “If  other,  please  specify.”  The  “Don’t 
Know  or  Not  Applicable”  response  usually  included  with  standard  response  scale  items  is 
another  way  of  being  exhaustive. 


Requesting  Tenure  in  Years  is  Easy  on  an  Automated  Survey.  The  years-on-the-job 
example  points  out  an  advantage  of  automated  surveys.  You  would  not  have  to  be  concerned 
about  these  issues  and  could  easily  get  a  more  precise  measure  of  years  (or  even  years  and 
months).  If  years  is  all  you  want,  say  “Approximately  how  many  years  have  you  worked  for 
(your  organization)?  (Round  up  to  the  nearest  year.)”  Then,  provide  a  space  where  the 
respondent  types  in  the  years. 

The  years  example  is  a  straightforward  request  that  most  respondents  would  handle 
correctly.  However,  beware.  If  you  ask  respondents  to  key  in  a  more  elaborate  numerical 
response,  the  request  may  be  executed  incorrectly  and  be  hard  to  interpret.  For  instance,  asking 
people  for  a  longer  set  of  numbers  (for  example,  the  numerical  designation  of  their  job  series,  a 
training  course  identifier,  or  their  social  security  number)  may  result  in  accidental  error. 


An  educational  attainment  item  I  reviewed  provides  another  example  of  an  item  that 
needed  more  exhaustive  and  mutually  exclusive  options.  Consider  the  following  item; 


Educational  level: 

A.  Less  than  high  school 

B.  High  school 

C.  1  -  3  years  of  college 

D.  Four  years  of  college—BS  degree 

E.  Master’s  degree  and  above 


This  item  uses  “less  than”  and  “and  above”  to  cover  extreme  cases,  but  it  leaves  out  categories  in 
between.  For  instance,  some  high  school  with  no  high  school  completion  is  not  covered.  There 
is  no  category  for  someone  who  did  not  finish  high  school,  but  who  did  obtain  a  GED.  What 
about  those  who  obtained  an  Associate’s  degree  from  a  junior  college?  What  about  those  who 
attended  a  vocational  school?  Option  “D”  is  vague.  Does  it  mean  four  years  for  those  who  had  a 
bachelor’s  degree,  or  four  years  of  college  with  or  without  a  bachelor’s  degree?  “BS  degree” 

(for  Bachelor  of  Science)  may  not  be  understood,  and  it  may  be  too  specific.  For  instance,  how 
would  a  person  with  a  Bachelor  of  Arts  degree  respond?  These  problems  can  be  corrected  by 
asking  for  highest  level  of  education,  increasing  or  refining  the  options  and  either  decreasing  the 
specificity  of  degrees  obtained  or  being  flexible  (by  using  i.e.,  or  etc.)  in  indicating  the  degrees 
obtained.  The  stem  would  be  better  phrased  less  abruptly  as  a  sentence.  For  example,  the 
following  item  was  used  on  the  San  Antonio  Air  Logistics  Center  Career  Planning  Survey.  I 
had  first  refined  the  original  stem  and  expanded  the  options.  Pretest  participants  suggested 
adding  options  E  &  H. 
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What  is  the  highest  level  of  education  you  have  attained? 

A.  Less  than  high  school 

B.  Some  high  school 

C.  High  school  graduation  or  GED 

D.  Technical,  vocational  or  business  school. 

E.  Some  college,  but  less  than  two  years 

F.  Two  years  of  college,  no  degree 

G.  Associate’s  degree 

H.  Three  years  of  college 

I.  Four  year  college,  no  degree 

J.  Bachelor’s  degree  (BA,  BS,  etc.) 

K.  Some  graduate  school,  no  degree 

L.  Master’s  degree  (MS,  MSW,  MBA,  etc.) 

M.  Doctoral  degree  (Ph.D.  JD,  MD,  DDS,  etc.)  or  above. 


In  this  example,  the  “M”  option  was  not  included  on  the  Logistics  Center  survey.  The  SA-ALC 
survey  ended  with  “L.  Master’s  Degree  or  above.”  How  specific  you  get,  and  at  what  level, 
depends  on  the  characteristics  of  your  target  population  and  your  information  needs.  If  members 
of  your  population  have  had  considerable  formal  education,  you  might  combine,  for  example,  the 
first  three  options.  You  might  also  cover  the  college  years  with  fewer  categories.  In  the  example 
above  “or  above”  was  included  in  response  “M”  to  cover  postdoctoral  work.  Perhaps  an  “Other” 
option  should  have  been  included. 

This  example  may  have  too  may  options  and  is  not  presented  here  as  the  only  education 
item  to  use.  Use  your  own  judgment  concerning  the  broadness  of  your  categories,  but  keep  them 
mutually  exclusive  and  exhaustive. 

Keep  Items  Neutral:  It  is  very  important  to  avoid  introducing  bias  into  your  item  stems 
or  responses  to  lead  respondents  to  a  desired  response.  But  keeping  items  neutral  does  not  mean 
they  are  to  be  completely  neutral.  As  Nunnally  (1978,  p.  605)  states,  “There  is  no  place  for  truly 
neutral  items  in  (Likert  type)  scales.”  Rather,  items  are  usually  mildly  positive  or  negative  since 
respondents  need  a  non-neutral  statement.  The  issue  is  that  respondents  should  convey  what 
they  think  or  feel  and  not  be  led  to  your  desired  response  (if  you  have  one).  Thus,  you  would  not 
write  an  item  stem  such  as  this: 


Do  you  favor  building  new  prisons  to  make  up  for  the  disastrous  policies  of  the  (whomever) 
administration  which  was  soft  on  crime? 


Likewise,  you  would  not  use  poorly  ordered  or  asymmetrical  response  options  like: 


Definitely  yes 
Probably  yes 
Yes 
No 


Doing  so  would  suggest  you  favor  more  prisons,  are  contemptuous  of  the  former  administration, 
and  desire  a  “yes”  response.  As  Bradbum  and  Sudman  (1988)  note,  such  loaded  questions  are 
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sometimes  found  on  pseudo-surveys  sent  out  by  special  interest  groups  to  increase  donations. 
They  are  persuasive  communication,  not  good  survey  items. 

Don’t  Mix  Apples  and  Oranges:  Sometimes  two  issues  are  covered  in  a  single  item.  Avoid 
this.  For  example,  consider  the  following  item: 


I  am  satisfied  with  upward  and  downward  communication  in  my  organization. 


A  person  may  be  happy  with  downward  communication  (because  top-down  management  may 
communicate  well  with  staff).  However,  he  or  she  may  believe  management  stifles  upward 
communication  by  adhering  too  strictly  to  the  chain  of  command,  or  by  preventing  easy  access  to 
senior  leaders.  Thus,  the  item  would  be  better  phrased  as  follows: 


How  satisfied  are  you  with: 

upward  communication  in  your  organization, 
downward  communication  in  your  organization. 


Note  that  the  phrase  that  distinguishes  between  two  otherwise  identical  items  is  highlighted  for 
emphasis.  These  rather  general  items  could  be  supplemented  with  others  tapping  potential 
specific  problems  with  organizational  communication. 

Getting  rid  of  redundancy  is  desirable.  If  the  items  above  are  with  several  other 
satisfaction  items  where  “in  your  organization”  does  not  always  apply,  then  leave  them  as  is.  If 
the  “in  your  organization”  phrase  applies  to  every  satisfaction  item,  it  can  be  extracted  from  the 
stem  and  included  in  the  lead  in,  as  follows: 


In  your  organization,  how  satisfied  are  you  with: 
upward  communication, 
downward  communication, 
your  immediate  or  first-line  supervisor, 
your  second-line  supervisor, 
your  senior  leadership, 
the  authority  you  have, 
the  autonomy  you  have, 
your  opportunities  for  career  progression, 
your  opportunities  to  influence  policies  affecting  your  job. 
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If  the  focus  on  “your  organization”  was  made  clear  in  introductory  comments/instructions,  the 
phrase  can  be  left  out.  Note  that  topics  which  are  almost  the  same,  but  slightly  different  (e,g., 
supervisor  level  and  authority/autonomy),  were  highlighted  for  emphasis.  Note  also  that  the 
terms  first-line  supervisor  and  second-line  supervisor  may  be  misunderstood.  Thus,  first-line  is 
also  defined  as  immediate;  by  implication,  second-line  is  distant. 

Here  is  an  item  mixing  trust  and  respect,  which  are  different  constructs: 

My  supervisor: 

has  earned  my  trust/respect. 

You  can  trust  someone  (not  to  betray  you),  but  you  may  not  respect  him  or  her.  For  instance,  the 
person  may  stick  up  for  you,  earning  your  trust.  But  the  same  person  may  have  other  qualities 
you  do  not  respect.  For  example,  he  or  she  may  be  inconsiderate  or  unethical.  Thus,  the 
following  revision  is  recommended: 

My  supervisor  has  earned: 
my  trust, 
my  respect. 

Note  that  two  things  were  done  in  this  example:  a  multiple  issue  item  was  broken  into 
two  items  and  redundancy  in  the  stem  (has  earned)  was  moved  to  the  lead-in  phrase.  This  would 
not  be  possible,  however,  if  “has  earned”  was  just  one  of  many  phrases  used  in  conjunction  with 
“my  supervisor.”  Note  that  these  items  are  phrased  in  a  way  to  give  only  a  general  idea  of  the 
degree  of  trust  and  respect.  If  more  detail  is  desired,  specific  items  could  be  included  to  tap  why 
the  supervisor  is  trusted  or  respected  (or  not). 

If  you  did  not  want  to  use  a  lead-in  phrase,  you  could  say: 

I  trust  my  supervisor. 

I  respect  my  supervisor. 

Again,  a  word  or  phrase  distinguishing  between  two  otherwise  identical  stems  was  highlighted. 

Another  item  from  the  same  survey  which  mixes  two  things  is: 

My  supervisor: 

believes  in  and  practices  quality  techniques. 

Belief  and  action  are  different.  The  supervisor  could  believe  in  quality  techniques  (as  in  Total 
Quality  or  Quality  Air  Force).  But  he  or  she  may  not  practice  them.  Even  though  this  will  result 
in  two  items,  the  following  alternatives  are  suggested: 
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My  supervisor: 

believes  in  the  quality  philosophy, 
uses  quality  techniques  at  work. 


Don’t  Be  Vague;  Tap  Directionality:  In  reviewing  the  Supervisor’s  Feedback 
Survey  for  the  San  Antonio  Air  Logistics  Center,  I  considered  the  following  item  too  vague: 


My  supervisor: 

provides  leadership  to  improve  communications  with  customers. 


What  is  the  nature  of  the  leadership  that  improved  communication  with  customers?  If  you  have 
a  hunch  what  it  is,  state  it.  If  it  could  come  from  several  actions,  include  multiple  items  to  tap 
these  facets.  One  suggested  alternative  is: 


My  supervisor: 

encourages  me  to  be  in  frequent  contact  with  my  customers  so  I  can 
respond  to  their  needs. 


Note  that  this  item  may  be  longer  than  necessary.  Is  the  “so  I  can  respond  to  their 
needs”  really  needed?  Ask  yourself  these  kinds  of  questions  as  you  refine  items. 

Let’s  criticize  an  item  from  the  author’s  own  Objective  Medical  Group  Survey: 

Since  OMG  implementation,  how  satisfied  have  you  been  with: 

the  autonomy  you  have  to  do  your  job  in  the  way  you  see  fit. 


Responses  to  this  item  would  tell  you  that  a  satisfied  person  was  pleased  with  the  autonomy  he 
or  she  had.  But  you  would  not  know  if  a  dissatisfied  person  wanted  more  or  less  autonomy.  Do 
not  assume  all  people  want  more  autonomy.  However,  if  you  used  the  following  statement  with 
an  agreement  scale,  you  would  tap  directionality,  knowing  that  if  they  agreed  with  the  statement 
they  wanted  more  autonomy. 


I  do  not  have  the  autonomy  I  need  to  do  my  job  in  the  best  possible  way. 


Consider  this  item  proposed  for  inclusion  in  the  San  Antonio  Air  Logistics  Center 
Career  Planning  Survey: 


Would  offering  a  separation  incentive  change  your  mind  about  retiring?  (yes/no) 


Now  consider  this  final  item,  clustered  with  similar  items  using  a  willingness  scale: 
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How  willing  would  you  be  to: 

take  early  retirement  for  a  cash  incentive  up  to  $25,000,  depending  on  grade  and  years  of 
service. 


In  the  original  item,  the  person  responding  was  given  no  idea  of  the  incentive’s  magnitude  and 
would  be  unable  to  make  an  informed  decision.  The  revised  item  is  more  specific.  However,  it 
may  be  flawed  by  not  providing  the  range  of  the  incentive  or  by  not  specifying  the  actual 
incentive  or  formula  for  calculating  it. 

Include  Negative  Items:  I  highly  recommend  negative  items.  Be  aware  that  corporate 
customers  may  frown  on  negative  items,  sometimes  claiming  they  may  foster  negative  results. 
This  is  doubtful  since  including  them  conveys  your  honesty  and  sensitivity.  Include  a  mix  of 
positive  and  negative  items  for  at  least  four  reasons: 

1)  it  breaks  the  respondent’s  tendency  to  answer  questions  in  the  same  way  (acquiescent 

response  set); 

2)  it  conveys  that  you  are  not  trying  to  put  a  “positive  spin”  on  the  results; 

3)  it  lets  respondents  know  you  are  sensitive  to  troubling  issues  and  aware  of  sources  of 

dissatisfaction,  and 

4)  it  is  likely  to  reduce  response  bias. 

Nunnally  (1978)  recommends  that  item  wording  should  be  mildly  negative  or  mildly 
positive  since  extreme  wording  tends  to  decrease  response  variance.  He  also  suggests  that  the 
mix  of  positive  and  negative  items  be  about  half  and  half.  Although  it  is  reasonable  to  have  such 
a  balance,  it  has  been  the  author’s  practice  to  include  fewer  negative  items  than  positive  ones.  In 
reviewing  the  attitude  measures  of  other  authors,  this  seems  to  be  a  common  trend.  Of  course, 
the  author’s  tendency  to  include  fewer  negative  items  may  be  a  conditioned  response  based  on 
his  awareness  of  client  sensitivity  to  such  items.  In  some  instances  it  has  been  the  result  of  client 
refusal  to  use  such  items. 

Concerning  the  issue  of  wording  strength,  it  has  been  the  author’s  practice  to  word  some 
positive  and  negative  items  somewhat  strongly  to  accurately  capture  sentiments  identified 
through  interviews  prior  to  survey  development.  This  lets  respondents  know  you  are  in  touch 
with  the  issues  and  aware  of  their  thoughts  and  feelings.  But  the  admonition  to  keep  the  positive 
or  negative  intensity  of  items  mild  is  generally  sound.  Intensity  is  captured  by  where  along  the 
response  scale  continuum  respondents  select  their  responses.  For  instance,  if  “Strongly  Agree” 
is  selected  in  response  to  a  mildly  negative  item,  the  implication  is  that  the  person  has  strong 
views  on  the  issue.  He  or  she  would  probably  have  phrased  the  issue  more  negatively  than 
indicated  by  the  item  stem,  if  asked  spontaneously  about  the  issue.  Keep  in  mind  that 
personality  differences  which  affect  responding  may  also  apply  here.  Some  people  seldom,  if 
ever,  select  the  extreme  responses,  even  if  they  have  strong  thoughts  or  feelings  about  the  issues 
expressed.  They  may  also  tone  down  their  strong  views  if  asked  about  them. 
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As  mentioned  above,  I  have  sometimes  targeted  items  about  specific  issues  for  positive 
or  negative  wording.  However,  the  decision  to  make  an  item  positive  or  negative  should  usually 
be  random.  As  colleague  Dr.  Walter  Albert  cautions,  to  avoid  biasing  responses,  this  approach  is 
preferable  to  the  practice  of  basing  the  decision  on  how  you  believe  most  respondents  would  feel 
about  an  item.  Some  negative  items  I  recommended  for  inclusion  in  the  Supervisor’s  Feedback 
Survey  used  at  the  San  Antonio  Air  Logistics  Center  are  provided  below. 


My  immediate  supervisor; 

only  plays  “lip  service”  to  quality. 

likes  to  micromanage  how  I  do  my  work. 

doesn’t  give  me  the  autonomy  I  need  to  do  my  job  as  I  see  fit. 

doesn’t  give  me  the  authority  I  need  to  do  my  Job. 

cares  more  about  pleasing  superiors  than  looking  after  subordinates. 

doesn’t  stand  up  for  his  or  her  subordinates. 


Note  that  the  phrasing  of  items  is  colloquial;  “doesn’t”  is  used  instead  of  “does  not.” 

Several  negative  items  were  included  on  the  OMG  Survey.  Some  examples  which  used 
a  common  lead-in  follow: 


With  implementation  of  the  OMG  structure: 

providers  and  care  givers  have  less  time  to  see  patients. 

my  administrative  workload  has  increased. 

my  facility  is  top  heavy  with  too  many  layers  of  management. 

I  am  less  clear  about  my  job  duties. 

I  am  less  clear  about  what  policies  to  follow. 

there  are  more  jobs  to  do  with  the  same  number  of  people  to  do  them. 

first  sergeants  have  too  little  involvement  in  the  well  being  of  enlisted  personnel. 

career  development  issues  are  more  difficult  to  deal  with  than  in  the  past. 


Here  are  some  negative  items  from  the  Objective  Medical  Group  Survey  which  do  not 
use  a  common  lead-in  phrase: 


It  doesn’t  make  sense  to  break  a  functional  medical  facility  into  several  squadrons  or  flights. 
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Persons  in  one  specialty  should  not  supervise  those  in  another  specialty. 

The  OMG  structure  cannot  be  implemented  successfully  without  providing  additional  personnel. 

It  is  a  disservice  to  some  of  our  best  people  to  assign  them  to  the  group  staff. 

The  following  negative  items  were  proposed  for  the  USAF  Financial  Management 
Career  Survey.  All  but  the  last  items  use  an  agreement  scale. 

I  am  unwilling  to  do  what  it  takes  to  get  promoted  in  the  financial  management  career  field. 

In  my  organization,  people  get  promoted  by  flattering  their  boss. 

In  my  organization,  people  get  promoted  through  aggressive  self-promotion. 

I  seldom  receive  unsolicited  information  about  training  opportunities. 

People  in  my  organization  are  seldom  informed  why  they  weren’t  selected  for  promotion. 
Getting  promoted  is  not  that  important  to  me. 

Performance  evaluations  are  so  political  or  inflated  that  they  are  practically  useless. 

For  what  reasons  would  you  not  want  a  promotion?  Select  all  that  apply. 

I  don’t  care  about  greater  recognition 
I  don’t  want  more  responsibility 

I  don’t  care  about  having  more  influence  in  my  organization 
I  don’t  want  to  be  promoted  out  of  my  profession 
I  like  what  I  am  doing 
I  don’t  want  to  leave  my  current  boss 
I  don’t  want  to  leave  my  current  organization 
I  don’t  want  to  enforce  higher  management  directives 
I  don’t  want  to  supervise  others 

I  wouldn’t  have  the  freedom  to  supervise  or  motivate  others  my  way 

I  would  have  to  move 

For  family  reasons 

I  am  about  to  retire 

Not  applicable 

Other  (specify) 


On  Organizational  Surveys,  Include  Some  General  Climate  Items:  When  they  desire 
to  conduct  organizational  surveys,  clients  often  focus  on  specific  concerns  such  as  organizational 
structure,  career  management/planning,  quality  progress,  customer  satisfaction,  quality  of  life,  or 
compressed  work  schedule.  Even  if  you  are  not  conducting  an  “organizational  climate”  survey, 
consider  including  a  few  items  on  organizational  climate  issues  to  monitor  how  well  your 
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organization  is  doing  on  key  climate  issues  like  communication,  satisfaction  with  various  facets 
of  the  job,  participation  in  decision  making,  autonomy,  innovation,  and  the  value  placed  upon 
human  resources.  The  following  climate  items  were  recommended  for  inclusion  in  the  USAF 
Financial  Management  Career  Survey  mentioned  above. 

I  trust  my  supervisor  to  act  in  my  best  interest. 

I  am  consulted  when  decisions  are  made  which  affect  me. 

I  could  do  my  job  better  if  I  had  more  autonomy. 

I  can  easily  talk  to  people  at  other  organizational  levels  and  in  other  departments. 

In  my  organization,  problems  or  disagreements  are  dealt  with  effectively  rather  than  “swept 
under  the  rug.” 

In  my  organization,  management  decisions  are  made  in  secret  and  then  sprung  unexpectedly  on 
the  workforce. 

I  have  to  rely  on  the  grapevine  (rumors)  to  find  out  what  is  going  on  in  my  organization. 

The  well  being  of  all  employees  or  AF  members  is  a  high  priority  in  my  organization 
People  in  my  organization  are  treated  with  respect. 

Creativity  is  rewarded  in  my  organization. 

I  would  like  to  participate  in  alternative  time/place  work  arrangements  such  as  flextime, 
compressed  work  schedule  or  telecommuting. 

I  am  satisfied  with  my  supervisor. 

I  am  satisfied  with  my  job. 

I  am  satisfied  with  the  financial  management  career  field. 


This  is  only  a  small  set  of  general  organizational  climate  items.  It  is  not  meant  to  be  exhaustive. 
In  context  with  other  similarly  formatted  items,  these  would  need  to  be  shortened  and  cleaned 
up.  For  Instance,  the  last  set  of  satisfaction  items  could  be  rephrased  to  have  a  “How  satisfied 
are  you  with”  lead  in.  In  addition,  the  common  phrase  “in  my  organization,”  used  in  several  of 
the  items,  could  be  extracted  and  used  as  a  lead  in. 

Include  Items  Only  On  Topics  Your  Client  Can  Do  Something  About:  When  you 
develop  items,  ask  yourself  this  practical  question:  can  the  people  who  will  use  the  data 
reasonably  have  an  impact  on  this  issue?  If  the  answer  is  no,  think  twice  about  including  the 
item.  Of  course,  you  may  anticipate  doing  something  about  the  problem  in  the  future,  or  you 
may  arrange  for  someone  else  with  authority  to  address  the  issue.  However,  if  your  client  has  no 
chance  of  effectively  rectifying  a  problem,  you  are  wasting  your  time  and  the  time  of  survey 
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participants  by  bringing  it  up.  You  will  also  raise  false  expectations  by  giving  the  impression  the 
problem  will  be  resolved. 

Several  items  were  deleted  from  the  Objective  Medical  Group  Survey  based  on  this 
principle.  For  instance,  the  following  item  was  deleted  because  it  was  covered  by  personnel 
regulations  beyond  the  AF  Surgeon  General’s  control: 


With  implementation  of  the  OMG  structure,  it  is  harder  to  get  General  Officer  signatures  on 
performance  appraisals. 


Likewise,  the  following  item  concerning  alternative  care  was  deleted  because  the  change 
it  represented  could  only  be  brought  about  through  Congressional  action:  (Of  course,  allopathic 
physicians-e.g.,  those  who  are  trained  in  and  practice  traditional  American-style  medicine—may 
also  have  been  disinterested  in  providing  these  services.) 


The  OMG  structure  of  my  medical  facility  should  include  alternative  forms  of  care  (such  as 
chiropractic  or  homeopathy). 


Keep  Items  Relevant  to  Your  Purpose,  Especially  if  Very  Intrusive:  Let’s  say  you 
must  include  very  intrusive  items  on  a  background  questionnaire  to  screen  people  for  sensitive 
positions.  For  example,  you  may  include  questions  about  drug  use.  Questions  about  current  use, 
or  use  in  the  recent  past,  may  be  reasonable  given  the  sensitive  nature  of  the  job  and  because 
behavior  in  the  present  and  recent  past  is  most  predictive  of  future  behavior.  However,  in  most 
instances  it  would  be  inappropriate  from  a  privacy  invasion  viewpoint  and  irrelevant  and  invalid 
from  a  predictive  viewpoint  to  probe  about  such  use  years  ago.  Respondents  may  decline  to 
answer  such  intrusive  questions,  or  you  may  get  misleading  answers,  even  to  questions 
concerning  current  or  near-term  behavior.  As  Koral  (1988,  p.  57)  warns  after  summarizing  legal 
issues  related  to  privacy,  “Don’t  get  information  you  don’t  really  need-snooping  can  get  you  in 
trouble,  and  having  information  that  is  not  really  necessary  can  get  you  in  trouble  if  you  use  it.” 
The  principle  of  not  asking  irrelevant  questions  applies  regardless  of  whether  they  are  very 
intrusive.  Shorten  your  survey  by  excluding  irrelevant  items. 

Make  Objectionable  Items  More  Acceptable:  In  Chapter  4, 1  recommends  limiting  or 
avoiding  objectionable  or  overly  intrusive  items.  He  does  acknowledge  they  should  be  included 
sometimes  since  controversial  issues  need  to  be  understood.  Dillman  (1978,  pp.  105-108)  and 
Salant  and  Dillman  (1994  p.  96)  and  Mael,  Connerley  and  Morath  (1996)  provide  suggestions  for 
making  objectionable  items  more  acceptable.  These  suggestions,  and  others  from  the  author,  are 
discussed  below. 

Dillman  and  Salant  and  Dillman  cite  that  asking  for  total  family  income  or  company 
profit  are  topics  many  people  consider  objectionable.  (As  mentioned  on  page  20,  Bradbum  et.  al 
also  found  income  questions  to  be  threatening.)  Rather  than  asking  for  specific  total  income  or 
profit,  they  suggest  providing  response  options  which  are  ranges  of  income.  For  instance,  you 
could  begin  and  end  with  “Less  than”  and  “More  than”  statements  and  have  ranges  of  five  or  ten 
thousand  dollars  (or  more).  They  recommend  you  do  not  make  your  categories  too  narrow  or 
people  may  think  you  are  trying  to  get  a  precise  income  figure  and  will  not  answer. 
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Dillman  and  Salant  and  Dillman  also  suggest  that  objectionable  content  be  placed  in  a 
balanced  context  to  soften  its  impact  and  to  convey  that  it  does  not  reflect  the  data  gatherer’s 
own  view.  Salant  and  Dillman  use  a  welfare  example.  Dillman  uses  an  example  of  an  item 
expressing  a  strong  negative  view  toward  the  church  as  a  “parasite  on  society,”  presumably 
because  churches  are  exempt  from  taxation.  This  is  a  strongly  worded  negative  item  that  many 
people  would  find  objectionable  since  Americans  typically  have  a  high  regard  for  religious 
institutions.  Dillman  suggests  the  addition  of  a  positively  worded  item  about  the  topic  as  a 
counterbalance.  He  recommends  a  lead-in  acknowledging  that  people  have  different  opinions 
about  the  role  of  the  church  in  society.  Notice  that  the  simple  lead-in  shown  below  is  neither 
negative  nor  positive.  However,  it  is  dated  and  implicitly  equates  organized  religion  with 
Christianity  through  the  use  of  the  word  “church.”  Today,  such  questioning  might  relate  to  the 
political  role  of  religious  groups  and  be  broadly  worded  so  anyone  can  answer,  regardless  of 
their  religious  views  or  affiliations. 

For  example,  the  following  lead-in  could  be  used  followed  by  several  items  using  an  agreement 
scale: 


Some  people  believe  that  religious  group  members  should  be  politically  active  to  help  change 
society.  Other  people  believe  they  should  focus  on  the  spiritual  needs  of  their  members.  In  the 
items  which  follow,  we  are  interested  in  your  views  on  this  issue. 


The  phrasing  of  this  lead-in  is  probably  not  offensive.  However,  this  assumption  should  be 
pretested  (see  Pretest  Your  Final  Draft  on  page  65).  The  question  is  broader  and  applicable  to 
more  people  than  the  example  concerning  the  church  provided  above.  It  can  be  answered  easily 
by  anyone  regardless  of  their  religion  and  religious  beliefs. 

Mael,  Connerley  and  Morath  ( 1 996)  discovered  that  specific  questions  about  religion 
(frequency  of  attendance  and  observing  specific  holidays)  were  considered  invasive  in  a  biodata 
inventory  for  job  selection.  They  also  discovered  that  a  question  about  specific  political  party 
membership  was  considered  veiy  invasive  in  the  same  context  while  a  question  about  being  a 
political  volunteer  was  less  so.  They  recommend  general  activity  questions  or  items  be  used 
which  avoid  specifics  about  denomination  or  membership.  To  that  the  current  author  adds  the 
recommendation  that  respondents  be  asked  their  attitudes  toward  political,  religious  (or  other) 
organizations  and  their  members,  rather  than  direct  questions  about  membership. 

Dillman  also  explains  that  the  most  difficult  items  to  write  are  those  which  could  lead  to 
respondent  condemnation  or  incrimination.  He  suggests  an  easy  way  to  make  such  items  less 
objectionable:  Rather  than  asking  people  about  themselves  and  their  behaviors  or  controversial 
views,  solicit  their  attitudes  toward  others  who  engage  in  such  activities  or  who  hold  such  views. 
You  could  also  ask  general  attitudes  toward  the  topic.  If  you  must  include  items  about  the 
respondent’s  own  behaviors  or  views,  Dillman  recommends  that  you  lead  up  to  them  by  placing 
them  after  general  attitude  items  (not  tied  to  the  respondent)  on  the  same  topic. 

You  may  be  able  to  get  indirectly  at  racist  or  sexist  attitudes  by  asking  people  about  the 
racism  or  sexism  of  those  with  whom  they  work  or  otherwise  associate.  However,  their  attitudes 
may  differ  from  those  around  them,  especially  at  work. 
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You  could  indirectly  assess  the  threat  of  survey  content  by  asking  how  threatened  other 
people  would  be  by  certain  questions,  as  Bradbum  et  al.  (1979)  did  (see  page  20). 

Sieber  (1992,  pp.  58-59)  also  provides  a  strategy  for  lessening  risk  to  respondents  and 
encouraging  truthful  answers  to  sensitive  items,  especially  those  involving  possible  illegal 
behavior.  It  allows  questions  to  be  answered  truthfully  or  untruthfully  based  on  the  role  of  a  die, 
and  then  the  researcher  calculates  the  estimated  true  proportion  of  people  who  actually  engaged 
in  the  behavior  under  study.  She  cautions  that  the  procedure  should  only  be  used  where 
“necessary  and  appropriate”  (presumably  where  distrust  and  apprehension  are  high).  However, 
she  cautions  that  this  tactic  may  backfire  by  arousing  suspicion. 

Inadvertent  Bias 

The  discussion  of  item  writing  and  scale  construction  brings  to  mind  another  issue: 
inadvertent  bias.  Responses  may  be  inadvertently  biased  by  many  factors:  how  items  are 
worded,  the  exclusion  of  negative  items,  respondents’  personalities  and  personal  constructions  of 
reality,  and  their  habituated  style  of  responding.  For  example,  just  like  some  people  are  reluctant 
to  criticize,  or  refrain  from  displaying  emotion,  some  respondents  refrain  from  using  the  full 
range  of  a  standard  response  scale.  Respondents  can  also  be  lulled  into  an  “acquiescent  response 
set”  whereby  they  respond  in  a  similar  fashion  to  multiple  items. 

Another  source  of  inadvertent  bias  is  “social  desirability  bias,”  that  is,  the  tendency  of 
respondents  to  tell  you  what  they  think  you  want  to  hear.  If  they  are  consciously  motivated  to 
evade  a  threatening  item,  this  would  not  constitute  inadvertent  bias  (but  it  would  still  be  a 
problem  for  those  interested  in  accurate  data).  Inadvertent  bias  would  occur  if  participants 
responded  in  a  more  positive  way  just  to  please  you  or  to  have  you  like  them.  The  desire  to 
please  is  probably  less  pronounced  on  surveys  than  in  interviews  since  the  survey  developer  is 
usually  not  present.  But  it  still  may  exist  since  we  are  trained  by  our  culture  and  expected  by  our 
family  and  organizations  to  please  others. 
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Chapter  8 


Comments  and  Return  Reminder 

An  important  part  of  a  survey  is  the  comments  section  at  the  end  and  comments  in  other 
places  where  they  would  be  desirable.  Comments  typically  provide  the  only  opportunity  for 
survey  participants  to  respond  in  their  own  words.  Thus,  spaces  for  comments  (and  brief 
narrative  items)  are  the  only  places  where  respondents  are  free  to  be  totally  unconstrained. 
Objective  response  options  can  be  justly  criticized  for  forcing  survey  participants  to  respond 
using  categories  and  words  of  the  survey  developer  which  may  not  fit  the  individual  survey 
participant. 

General  Comments 

Surveys  should  have  a  general  comments  section  at  the  end  which  provides  space  for  at 
least  a  page  of  optional  feedback  about  the  survey  or  the  topics  it  measures.  If  people  need  more 
space,  they  can  write  on  the  back  of  the  page  or  use  separate  sheets  of  their  own.  Do  not  put 
lines  in  this  space  if  your  survey  is  paper-and-pencil  since  people  write  in  all  sizes  and  shapes. 
You  could  use  an  introductory  lead-in  like  this: 


Surveys  do  not  measure  all  issues  that  may  concern  you.  If  you  wish,  provide  additional 
comments  on  (your  topic)  or  on  this  survey  in  the  space  below.  If  you  refer  to  a  specific  item, 
please  indicate  the  item  number. 


Note  that  although  this  is  a  general  comments  section  lead-in,  respondents  are  encouraged  to 
reference  a  specific  item  number  to  help  you  group  responses  later. 

A  comments  section  allows  people  to  express  themselves  in  their  own  words  on  the  topic 
of  interest  to  you,  or  on  any  topic  they  want  to  tell  you  about.  It  is  a  rich  source  of  additional 
information  that  sometimes  provides  you  with  your  best  information  on  how  to  tackle  a  problem. 
Typically,  comments  are  provided  by  about  40%  of  those  who  return  the  author’s  surveys.  They 
range  from  a  word  or  two  to  well  crafted  letters  several  pages  in  length.  A  brief  paragraph  is 
typical.  Some  people  take  the  comments  so  seriously  they  provide  their  name  and  phone  number 
and  ask  that  you  call  them.  If  they  request  this,  follow  through  to  increase  their  sense  of  being 
consulted  and  to  help  you  gain  additional  insights. 

Comments  sections  are  difficult  to  analyze.  You  can  do  a  content  analysis  (described  in 
more  detail  on  page  72)  where  you  identify  themes  and  subthemes  and  count  the  frequency  with 
which  they  are  mentioned.  When  reporting  back  to  your  customer,  you  can  even  quote  some  of 
the  most  representative  comments  on  critical  themes.  If  you  received  the  comments  from  an 
automated  survey  or  transcribed  them  to  a  disk,  you  can  do  key- word  searches  to  facilitate  your 
understanding  of  themes.  These  approaches  are  not  sophisticated  and  they  are  labor  intensive. 
However,  comments  are  rich  sources  of  understanding  even  if  all  you  do  is  read  them.  Once  you 
read  the  first  hundred  or  so  comments  pages  the  remainder  will  probably  be  redundant.  You  will 
likely  have  identified  all  the  major  themes. 

Two  cautions  about  comments  are  in  order.  First,  comments  are  more  often  negative 
than  positive.  Those  who  are  dissatisfied— those  who  management  may  want  to  dismiss  as 
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malcontents  and  muckrakers—are  more  likely  to  take  time  to  write  comments.  In  the  author’s 
experience,  this  has  led  some  managers  to  discount  them.  Encourage  management  to  take 
comments  seriously.  Perhaps  you  could  encourage  more  balanced  comments  by  adding  a 
sentence  like  this  to  your  comment  section  lead-in: 


Since  it  is  important  for  us  to  know  what  we  are  doing  right  as  well  as  wrong,  please  provide 
positive  and  negative  comments. 


You  could  even  have  respondents  categorize  comments  as  either  positive,  negative,  or  other. 

The  second  caution  pertains  to  confidentiality  and  non-attribution.  Comment  transcripts 
should  not  be  attributed  to  specific  individuals,  regardless  of  whether  they  are  the  object  or  the 
source,  except  if  the  reference  is  to  a  person  whose  identity  would  be  hard  to  conceal  (for 
example,  the  commander,  the  CEO,  the  department  chair,  etc.).  By-name  references,  especially 
of  a  negative  nature,  should  be  deleted  before  being  shared  with  management.  Thus,  if  a  person 
said  “I  am  upset  with  the  way  my  hospital  commander  treats  the  members  of  the  group  staff 
because  he  (she)  fails  to  keep  them  involved  in  decision  making,”  that  would  be  a  fine  comment 
to  share.  But  if  someone  said  “Colonel  so  and  so  is  a  (expletive  deleted)  who  cares  only  about 
himself  and  his  own  career  advancement,”  the  reference  to  the  officer’s  name  should  be  excised 
before  the  transcript  is  shared  with  management.  It  might  be  constructive  for  management  to 
know  the  frequency  and  nature  of  such  comments  (to  get  an  idea  of  the  quality  of  leadership),  but 
the  individual  who  is  the  object  of  the  comment  should  be  protected  since  the  accusation  is 
hearsay  and  may  be  without  merit.  There  is  probably  an  exception  to  this  general  rule,  and  this 
is  hypothetical  since  I  have  not  encountered  it  in  a  survey  comment.  If  serious  wrongdoing  is 
alleged,  it  may  need  to  be  investigated  further,  but  it  would  have  to  be  proven  and  the  rights  of 
the  accused  would  have  to  be  safeguarded. 

Sometimes  it  is  difficult  to  hide  the  identity  of  the  person  who  is  the  subject  of  a 
comment,  and  it  may  not  even  be  desirable.  People  in  executive  positions  can  easily  be 
identified  by  their  title.  They  need  to  be  made  aware  of  criticism  (in  a  helpful  way),  able  to 
accept  it  without  becoming  too  defensive,  and  able  to  grow  from  criticism  which  seems  fair.  If 
similar  criticism  comes  from  several  sources,  there  is  probably  truth  to  it. 

The  identity  of  the  provider  of  such  information  should  be  held  in  confidence,  and 
executives  themselves  should  be  shielded  from  negative  comparisons  with  their  peers.  The 
information  should  be  used  to  help  improve  executive  effectiveness  and  enhance  organizational 
functioning  rather  than  to  put  executives  down  in  front  of  their  colleagues. 

Item-Specific  Comments 

On  short  paper-and-pencil  surveys  to  a  small  group  of  people,  it  may  be  reasonable  to 
provide  space  for  brief  comments  to  each  item  or  to  selected  items.  However,  on  long  paper- 
and-pencil  surveys,  on  those  using  a  separate  optical  scan  sheet,  and  on  those  going  to  huge 
numbers  of  people,  this  would  be  less  feasible. 


Item-Specific  Comments  on  Automated  surveys.  Allowing  comments  to  every  item  is 
quite  feasible  on  automated  surveys  and  some  survey  software,  like  that  from  the  Armstrong 
Laboratory,  permit  it.  Use  of  this  option  is  encouraged.  Don’t  fear  getting  back  too  much 
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information  or  having  to  perform  complex  analyses  of  this  narrative  input.  Even  in  raw  form, 
these  comments  may  be  quite  helpful.  For  instance,  imagine  that  your  survey  is  going  to  people 
at  many  different  facilities  and  you  wish  to  use  your  survey  to  identify  needed  improvements  at 
each  facility.  If  objective  responses  to  a  particular  item  provide  a  general  indication  of  a 
problem  at  a  particular  site,  you  could  print  all  comments  to  that  item  made  by  people  at  that 
facility  to  get  richer,  more  specific  information  about  the  nature  of  the  problem.  Having  a 
manager  at  the  facility  read  these  site-specific  comments  would  not  be  too  burdensome. 


Thank  You  Message  and  Return  Reminder 

After  your  general  comment  section,  remember  to  thank  respondents.  Don’t  say 
something  abrupt  like  “The  survey  is  now  complete.”  Also,  remind  them  of  how  to  save  their 
data  and  exit  the  survey  if  it  is  automated  and  how  to  return  the  survey  to  whomever  will  analyze 
it.  On  a  paper-and-pencil  survey  using  a  separate  optical  scan  sheet,  your  thank  you  message  and 
return  reminder  could  be  as  simple  as  this: 


Thank  you  for  completing  this  survey.  Please  return  your  comments  page  and  answer  sheet 
(unfolded  and  unstapled)  to  your  local  survey  administrator  or  directly  to  the  following  address: 
(included  address). 
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Chapter  9 


Creating  a  Survey 

Assume  you  now  have  the  knowledge  to  put  a  survey  together.  Where  do  you  start? 
How  do  you  go  about  developing  items  and  formatting  them  and  the  other  parts  into  a  good 
survey?  In  this  chapter,  the  process  of  actually  putting  a  survey  together  is  discussed. 

Identify  Themes  You  Wish  to  Measure 

Don’t  just  start  writing  items.  First,  identify  the  themes  you  wish  to  measure.  You  can 
do  this  in  a  number  of  ways.  You  can  go  to  the  academic  literature,  your  customers,  or  to 
representatives  of  the  target  population  you  wish  to  survey.  Naturally,  you  can  use  all  of  these 
approaches  to  be  veiy  thorough. 

Consult  the  Literature  or  Scholars 

Academic  surveys  usually  contain  content  measuring  various  constructs  of  a  theoretical 
model  of  organizational  or  social  phenomena.  The  items  measuring  each  construct  of  the  model 
are  called  the  “operational  definition”  of  the  construct  and  are  at  first  conceptually  determined 
and  later  empirically  validated  (see  Quantitative  Inferential  Analyses  and  Results  Reporting  on 
page  71).  The  ideas  for  these  models  and  constructs  typically  come  from  the  literature, 
government  technical  reports,  similar  sources,  or  from  scholars  themselves. 

I  consulted  the  literature  when  I  studied  enlisted  personnel  retention  (Watson,  1986a, 
1986b)  with  the  United  States  Air  Force  Retention  Survey.  The  academic  literature  was 
extensively  reviewed  and  a  conceptual  model  of  turnover  and  retention  was  developed  for  testing 
which  contained  over  twenty  constructs  (themes).  Next,  items  were  developed  (or 
borrowed/adapted  from  other  sources)  to  measure  the  constructs.  In  the  course  of  this  research  it 
was  discovered  that  the  original  conceptual  operational  definitions  of  the  constructs  were  similar 
to,  but,  nonetheless  different  from,  the  empirically-determined  constructs.  That  is,  the  items  that 
were  summed  and  averaged  to  form  multiple-item  scales  to  measure  these  constructs  changed 
somewhat.  Construct  names,  and  the  items  used  to  measure  them,  were  slightly  different.  In 
addition,  the  final  empirical  model  had  fewer  variables  (constructs)  than  the  original  conceptual 
model.  This  is  what  typically  happens  in  research  involving  the  testing  of  a  theoretical  model. 

If  you  are  academically  inclined,  your  themes  become  constructs  or  variables  to  measure 
and  you  think  of  them  as  predictor  or  criterion  variables  to  test  in  a  model.  Thinking  in  this  way 
is  helpful  if  you  want  to  “determine  the  psychometric  properties”  of  your  survey  (this  typically 
means  to  determine  its  reliability  and  validity).  Notice  that  if  you  are  measuring  multiple 
constructs,  it  is  not  the  instrument  as  a  whole  that  gets  assessed  but  the  reliability  and  validity  of 
each  construct  it  measures. 

Remember  that  in  many  organizational  research  or  survey  consultation  applications 
elaborate  analyses  would  not  need  to  be  performed.  Descriptive  statistics  may  suffice.  See 
Quantitative  Descriptive  Statistics  and  Results  Reporting  on  page  70. 
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Consult  Those  Who  Requested  the  Survey 


You  can  consult  another  type  of  expert  outside  the  universities  and  not  published  in  the 
literature.  These  experts  are  your  customers  who  asked  you  to  develop  a  survey— the  executives, 
community  leaders,  or  others  in  authority  who  ask  you  to  investigate  an  issue.  Often  they  have 
considerable  knowledge  to  help  you. 


When  I  was  asked  to  study  physician  leadership  development  by  the  Air  Force  Surgeon 
General  and  the  Wilford  Hall  Commander,  he  became  a  member  of  the  Surgeon  General’s  Ad 
Hoc  Committee  on  Physician  Leadership  Development  and  was  asked  to  develop,  administer  and 
analyze  the  Physician  Leadership  Survey.  Being  on  the  committee  gave  him  direct  access  to 
the  senior  Air  Force  medical  leadership.  They  became  the  content  experts  and  I  was  able  to 
elicit  from  them  what  the  content  of  the  survey  should  be.  The  knowledge  I  gained  from  them 
was  confirmed  and  supplemented  through  interviews  with  physicians  at  Wilford  Hall  Medical 
Center. 

In  this  case,  simple  descriptive  analyses  were  performed.  They  were  sufficient  to 
provide  good  feedback  to  the  senior  leadership  (Watson,  1993).  For  an  academic  audience, 
inferential  analyses  were  later  conducted  and  reported  (Mueller  &  Watson,  1996). 


Consult  People  at  the  Grass  Roots 

Scholars  in  academia  or  management,  or  organizational  and  community  leaders,  are  not 
the  only  good  sources  of  survey  content.  People  in  different  roles,  in  different  groups,  or  at 
different  organizational  or  societal  levels  have  their  own  unique  perspective  on  issues.  The 
literature,  or  even  those  people  who  asked  you  to  build  your  survey,  should  not  be  the  only 
sources  of  information  for  identifying  themes.  They  may  not  even  be  the  best  sources.  An 
excellent  source  of  information  are  the  people  impacted  by  what  you  are  studying,  at  all  levels  of 
an  organization,  or  in  different  walks  of  life.  It  is  a  very  good  idea  to  interview  people  at  the 
grass  roots  as  part  of  the  process  of  survey  development.  You  can  get  excellent  ideas  for  survey 
content— the  issues  to  tap.  You  can  also  learn  how  to  write  your  survey  in  a  way  that  suits  the 
respondent  population.  As  Stewart  and  Shamdasani  (1990)  point  out,  interviews  can  help  you 
discover  how  respondents  talk  about  the  issues  of  interest,  explore  alternative  item  formats,  and 
examine  the  suitability  of  your  response  scales.  See  Appendix  J  for  brief  interview  guidance. 

For  more  complete  information  see  Watson  (1996). 

Are  you  in  marketing  research  and  evaluating  a  product  or  service?  If  so,  customers  who 
have  used  the  product  or  service  are  an  excellent  source  of  ideas  for  survey  content.  Such 
customers  are  likely  to  be  quite  familiar  with  what  you  are  studying,  know  what  they  like  or 
don’t  like  about  it,  and  have  suggestions  concerning  how  it  should  be  improved. 


When  I  was  asked  to  develop  the  USAF  Medical  Service  Objective  Medical  Group 
(OMG)  Survey,  I  went  to  the  grass  roots:  to  people  in  representative  Air  Force  medical 
facilities  who  lived  day-to-day  with  the  new  structure.  Individual  and  group  interviews  were 
conducted  using  an  unstructured,  permissive  focus-group  approach  to  interviewing  (see  Krueger, 
1988;  Watson  1997)  to  identify  themes.  I  didn’t  just  go  to  management  within  these  facilities. 
People  at  all  levels  of  the  organization  and  in  many  different  roles  were  interviewed.  After 


62 


interviews  in  five  locations  he  was  able  to  identify  over  thirty  themes  and  had  taken  about  100 
pages  of  notes.  These  themes  and  the  interview  notes  were  used  to  develop  370  attitudinal  items 
from  which  1 1 3  were  selected  and  refined  for  use  in  the  final  survey.  Note  from  this  example 
how  it  is  wise  to  develop  far  more  items  than  you  will  use.  From  this  large  set,  you  can  then 
select  and  refine  the  best  ones. 

Why  didn’t  I  just  go  to  the  literature  or  the  AF  Surgeon  General’s  staff  to  get  themes  and 
item  ideas?  Why  was  going  to  people  in  the  field  so  critical?  Naturally  what  little  literature  was 
available  on  the  OMG  structure  was  read,  and  the  Surgeon  General’s  staff  was  consulted.  In 
fact,  they  were  most  helpful  in  assisting  with  survey  refinement.  But  they  were  not  sought  as  the 
main  source  of  information  on  how  to  identify  themes  for  at  least  the  following  reasons: 

•  the  OMG  structure  had  recently  been  implemented  in  medical  facilities  and  there  was 
little  literature  available  on  it; 

•  the  Surgeon  General’s  staff  were  in  headquarters  geographically  separated  from  the 
medical  facilities  involved  in  OMG  implementation; 

•  the  Surgeon  General’s  staff  were  responsible  for  OMG  implementation  and  may  have 
had  a  personal  investment  in  seeing  it  carried  through,  and 

•  1  was  an  impartial  outsider  fi-om  a  respected  research  organization,  skilled  at 
interviewing  and  had  previously  established  a  reputation  in  the  Air  Force  medical 
community  of  being  trustworthy  and  discrete.  Therefore,  medical  personnel  were  more 
likely  to  open  up  to  me  about  their  concerns  than  they  may  have  been  to  members  of  the 
headquarters  staff. 


Explore  Using  Existing  Items  First 

Assume  you  already  identified  themes.  What  do  you  do  next?  Items  are  the  most 
important  part  of  your  survey,  so  start  with  them.  Don’t  just  start  writing  and  possibly  reinvent 
the  wheel:  Use  existing  items  if  they  are  available  at  little  or  no  cost  and  you  do  not  infringe  on 
copyright  laws.  If  you  are  a  Government  employee.  Government  surveys  are  a  good  starting 
place  since  they  can  be  used  by  others  in  Government.  Look  also  at  surveys  from  other  sources 
for  item  ideas  relevant  to  your  topic,  even  if  they  are  proprietary.  In  some  situations  you  may 
want  to  tailor  existing  items  to  your  situation.  Sources  I  frequently  consult  include  the  following 
works  by  scholars  at  the  Institute  for  Social  Research  at  the  University  of  Michigan:  Robinson, 
Athanasiou  and  Head  (1969),  Taylor  and  Bowers  (1972),  Robinson,  Rusk  and  Head  (1973)  and 
Robinson  and  Shaver  (1973).  These  references  are  dated  but  are  still  useful.  I  also  consult 
Robinson,  Shaver  and  Wrightsman  (1991),  a  more  more  recent  source  by  some  of  the  same 
scholars. 

In  social  science  research,  investigators  often  strive  to  use  multiple  item  scales 
developed  by  other  researchers  because  the  psychometric  properties  (especially  reliability  and 
validity)  of  these  instruments  have  been  established  across  many  samples.  Note  that  you  may 
not  be  able  to  change  a  proprietary  item,  and  if  you  do  change  items,  reliability  and  validity  may 
be  affected. 
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Generate  Initial  Item  Ideas 


After  you  have  identified  themes  you  wish  to  measure  and  have  explored  using  existing 
items,  begin  writing  items  (or  item  ideas  as  I  call  initial  crude  items)  to  measure  what  existing 
items  do  not  tap.  Concentrate  on  possible  item  stems  and  write  down  ideas  for  items  without 
concern  for  quality,  that  is,  write  whatever  comes  to  mind  about  the  theme.  Do  not  edit  or 
review  your  item  idea  yet.  Write  down  your  next  item  idea  and  keep  doing  this  until  you  have 
covered  all  aspects  of  the  theme  you  can  think  of.  If  more  item  ideas  come  to  you  later,  write 
them  down  immediately  before  you  forget  them.  Carry  a  few  index  cards  around  just  for  that 
purpose.  Do  not  concern  yourself  with  redundancy  or  worry  about  response  scales  this  early  in 
the  process  unless  you  have  to  write  items  that  will  use  item-specific  responses. 

The  recommendation  that  you  do  not  have  to  be  concerned  about  quality  at  first  may 
seem  odd,  but  it  is  the  only  way  you  will  generate  lots  of  potentially  good  items  without  stifling 
your  ideas.  At  this  early  stage,  you  are  brainstorming  and  should  expect  to  generate  two  to  three 
times  as  many  items  as  you  will  probably  use.  There  will  be  the  opportunity  later  to  refine  good 
items  and  get  rid  of  the  bad  ones. 

Cull  out/Refine  Your  Items;  Select  Scales 

After  you  have  amassed  more  items  than  you  expect  to  use,  critically  review  your  items. 
Cull  out  poor  or  redundant  ones  and  refine  the  rest.  (Note;  you  may  want  redundant  items  if 
you  plan  to  form  multiple-item  scales.  They  would  not  be  the  same-just  similar,  getting  at 
different  facets  of  the  same  theme  or  at  the  same  issue  in  slightly  different  ways.)  Ask 
colleagues  to  help  you.  Also,  begin  to  think  about  which  common  response  scale(s)  you  wish  to 
use  and  begin  to  cluster  items  by  scale. 

Some  surveys  use  multiple  scales  while  others  only  use  one.  The  agreement  scale  seems 
to  be  the  most  popular,  followed  by  the  satisfaction  scale.  Note  that  with  slight  rewording,  items 
can  be  modified  to  accommodate  different  scales.  The  same  wording  can  be  used  for  agreement 
and  extent  scales. 

Put  Together  an  Initial  Draft  Survey 

You  may  have  already  been  developing  and  refining  items  using  a  computerized 
authoring  system.  If  not,  this  is  the  time  to  begin  to  put  the  survey  together  in  a  way  that 
approximates  its  final  format.  After  you  refined  the  surviving  items  and  clustered  them  by  a 
common  scale,  if  applicable,  add  other  survey  components.  Develop  a  cover  page  (optional),  a 
cover  letter  (optional  but  recommended),  introductory  comments  and  directions,  and  background 
or  demographic  items.  Identify  items  that  should  logically  be  grouped  together  (i.e.,  other  than 
simply  on  the  basis  of  scale  used)  Companion  items  differing  only  by  a  word  or  a  phrase  would 
be  an  example.  Tentatively  identify  and  name  the  parts  of  your  survey.  Usually  background 
items  are  placed  before  more  general  attitudinal  items  as  Part  1.  I  recommend  this  approach. 
However,  some  authors  put  demographic  items  last.  Add  a  comments  section  after  your  items 
and  a  thank  you  and  return  reminder.  If  desirable  and  feasible,  provide  opportunities  for  item- 
specific  comments  after  each  item  or  after  selected  items.  Item-specific  comments  are  more 
feasible  on  automated  surveys. 
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Finalize  Your  Draft 


Share  your  draft  survey  with  your  colleagues  and  client.  Have  them  help  you  refine  it 
and  cull  out  additional  items.  This  is  usually  a  multiple-stage  process.  For  instance,  six 
iterations  were  used  to  develop  the  USAF  Medical  Service  Objective  Medical  Group  (OMG) 
Survey.  If  you  are  going  to  automate  your  survey  and  have  not  used  a  computer-based  authoring 
system  to  create  it,  this  is  the  time  to  transform  it  from  paper-and-pencil  to  automated  form. 

Pretest  Your  Final  Draft 

After  your  colleagues  and  client  have  looked  over  your  survey  and  helped  you  get  it  into 
final  shape,  pretest  it.  Have  your  colleagues  and  client  take  the  survey  and  record  how  long  it 
took  them  to  do  so.  On  the  basis  of  actually  having  taken  the  survey,  ask  them  to  offer 
suggestions  for  further  improvement. 

Having  colleagues  and  your  client  pretest  the  survey  is  good,  but  not  sufficient.  Have 
about  50  to  100  people  take  your  survey  who  are  representative  of  the  target  population  (your 
intended  respondents).  Time  how  long  it  takes  them  to  complete  it.  After  you  have  administered 
it,  interview  them.  Abbreviated  guidance  on  interviewing,  extracted  from  the  author’s 
Guidelines  for  Conducting  Interviews  (Watson,  1997),  is  provided  as  Appendix  J.  Probe  to 
see  if  the  your  items  and  directions  are  clear,  if  survey  length  is  acceptable,  and  if  any  items  are 
offensive  or  too  intrusive.  Ask  pretest  participants  if  they  believe  the  right  items  have  been 
asked.  Ask  them  also  to  suggest  item  modifications  or  additions.  Have  them  suggest  other  ways 
to  improved  your  survey.  Suggestions  concerning  the  questions  to  ask  in  a  pretest  interview  are 
provided  in  Appendix  K. 

Suggestions  for  improvement  are  not  a  sign  of  failure  on  your  part.  Be  open  to 
constructive  criticism  without  being  defensive.  It  would  be  unreasonable  to  think  you  could 
construct  a  flawless  survey  on  your  own.  You  are  not  the  subject  matter  expert.  These  people 
are  your  partners.  Listen  to  their  suggestions  and  include  those  which  appear  reasonable. 

Construct  Your  Final  Survey 

On  the  basis  of  the  feedback  you  receive  from  colleagues,  customers,  and  representatives 
of  your  target  population,  develop  a  final  survey,  paying  attention  to  detail.  Now  is  the  time  to 
make  sure  you  are  consistent  throughout,  that  all  typographical  and  spelling  errors  have  been 
eliminated,  and  that  the  survey  is  attractive  and  professional.  If  you  automate  your  survey,  make 
sure  it  works,  not  only  on  your  system  or  area  network  but  also  on  other  systems  and  networks. 
Make  sure  branching  occurs  as  planned  and  that  bugs  are  worked  out. 

Request  Approvals  or  Exemptions 

In  addition  to  approval  from  your  customer  and  your  own  organization,  you  may  need 
additional  approvals  or  exemptions  as  discussed  below. 

Headquarters  Air  Force  Personnel  Center  (HQ  AFPC)  Approval.  Air  Force 
personnel  who  are  conducting  a  survey  extending  beyond  their  local  organization  or  base  need  to 
coordinate  their  survey  with  the  Headquarters,  Air  Force  Personnel  Center  (HQ  AFPC).  This 
office  conducts  Air  Force  surveys,  protects  people  from  being  over  surveyed,  acts  as  survey 
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consultants  and  provides  quality  control  for  Air  Force  surveys.  Their  address  is  HQ 
AFPC/DPSAS,  550  C  Street  West,  Suite  35,  Randolph  AFB,  TX  78150-4737.  Their  phone 
number  is  (210)  652-5680  (Defense  Switching  Network  or  DSN  prefix:  487).  People  in  this 
office  can  help  you  with  all  aspects  of  survey  research  and  you  may  even  arrange  for  them  to 
conduct  a  study  for  you.  Typically,  however,  they  review  your  survey  and  provide  suggestions 
for  improvement.  Once  they  are  comfortable  with  it,  they  assign  a  survey  control  number  (SCN) 
with  an  expiration  date.  More  information  on  Air  Force  surveys  is  contained  in  Air  Force 
Instruction  36-2601 . 

If  you  are  interested  in  other  survey  resources,  Bradbum  and  Sudman  (1988,  Chapter  4) 
provide  a  good  list  of  professional  survey  organizations  in  government,  the  private  sector  and 
academia.  Although  the  list  is  not  exhaustive,  it  is  extensive. 

Office  of  Management  and  Budget  (OMB)  Clearance.  If  you  are  in  the  Government 
and  want  to  survey  respondents  who  are  not  Government  employees,  you  will  likely  need  to  get  a 
clearance  from  the  Office  of  Management  and  Budget  (OMB).  This  process  can  be  time 
consuming. 

Institutional  Review  Board  Exemption:  To  protect  subjects  from  possible 
mistreatment  in  experiments,  Institutional  Review  Boards  (IRBs)  have  been  created  in  the  Air 
Force  and  elsewhere  to  provide  ethical  review  of  proposed  research  and  oversight  of  research  in 
progress.  See  Sieber  (1992)  for  a  more  detailed  discussion  of  IRBs. 

Anonymous  surveys  in  which  names  or  other  personal  identifiers  are  not  requested  are 
usually  exempt  from  scrutiny  from  such  boards.  This  is  despite  the  fact  that  individual 
respondents  can  sometimes  be  identified  from  their  unique  combination  of  demographic  data. 
Although  Air  Force  anonymous  surveys  are  exempt.  Air  Force  researchers  need  to  formally  ask 
for  exemption  from  local  review  boards  such  as  the  Advisory  Committee  for  Human 
Experimentation  at  Brooks  Air  Force  Base.  The  use  of  human  subjects  in  research  is  discussed 
in  AL  Instruction  40-1  and  in  AL  Handbook  40-1. 
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Chapter  10 


Data  Analyses  and  Results  Reporting 
Prepare  to  Perform  Analyses 

Clean  Up  Optical  Scan  Sheet:  If  you  use  a  scan  sheet,  or  a  scannable  survey,  make 
sure  the  guidance  you  provided  concerning  how  to  mark  responses  in  a  machine-readable  way 
was  followed.  If  not,  correct  the  sheet  or  booklet,  or  accurately  transcribe  responses  onto  a  new 
form.  For  instance,  was  a  pencil  used?  Was  the  sheet  folded  or  stapled?  Are  the  ovals  or 
bubbles  filled  in  darkly  enough  to  be  optically  scanned  without  difficulty?  Are  there  stray  marks 
(if  so,  erase  them)?  Do  the  marks  wander  far  outside  the  ovals  (if  so,  erase  that  portion  which  is 
outside)?  Are  there  clear  signs  that  some  participants  did  not  respond  seriously  (see  Perform  a 
Veracity  Check  below)?  If  so,  do  not  include  data  from  those  sheets  or  surveys  in  your  analyses. 

Perform  a  Veracity  Check:  Regardless  of  whether  or  not  you  used  an  optical  scan 
sheet  or  a  scannable  survey  booklet,  you  need  to  identify  those  respondents  who  did  not  take 
survey  completion  seriously.  Most  people  who  do  not  want  to  take  your  survey  simply  do  not 
respond.  However,  some  do  not  take  the  survey  seriously  and  provide  responses  that  are  not 
trustworthy.  Naturally,  you  are  not  always  sure  if  this  is  the  case,  but  try  to  identify  these  people 
and  do  not  enter  them  in  your  data  base. 

How  would  you  do  this?  Either  visually  or  with  the  aid  of  a  computer,  identify 
respondents  for  whom  you  would  answer  yes  to  the  following  questions.  Did  some  participants 
respond  only  to  a  few  questions  and  then  stop?  Did  some  provide  a  repetitious  response  pattern 
like  zigzags?  Did  the  responses  of  some  people  vary  little  or  not  at  all?  Did  you  get  profanities 
about  you  or  your  survey  in  the  bubbles  of  the  answer  sheet,  or  scrawled  on  it  is  some  other  way? 
Did  some  respondents  provide  inconsistent  responses,  such  as  very  different  responses  to  nearly 
identical  items?  If  you  included  a  valid  “lie  scale,”  were  there  clear  indications  of  probable 
deception?  Did  some  respondents  fail  to  shift  their  response  pattern  when  they  came  upon 
negatively  worded  items? 

CATI  telephone  interviewing  technology  quickly  alerts  interviewers  about  inconsistent 
responding  while  the  interview  is  in  progress.  Thus,  the  interviewer  can  probe  to  clear  up  the 
inconsistency  and  determine  if  it  was  accidental  or  intentional. 

If  you  did  not  use  an  optical  scan  capability  or  computer  and  instead  transcribed 
responses  marked  on  the  survey  itself,  there  is  another  kind  of  veracity  check  to  perform.  This 
one  concerns  accuracy  rather  than  honesty.  Have  a  person  who  did  not  perform  the  original 
transcription  check  the  data  for  accuracy  since  transcription  is  monotonous  and  mistakes  could 
have  been  made. 

Scan  Sheets  and  Create  a  Data  File:  Create  a  data  file  suitable  for  performing  your 
analyses  if  it  has  not  already  been  done.  Also,  separate  data  suitable  for  qualitative  analysis  from 
data  suitable  for  quantitative  analysis.  For  example,  create  separate  files  for  comments,  for 
narrative  responses  to  items  and  for  data  fi-om  items  using  non-narrative  response  scales. 

Separate  and  save  narrative  comments  on  your  general  comments  page  or  comments 
pertaining  to  specific  items.  Save  these  on  paper,  in  your  computer,  or  on  a  disk  in  an  organized. 
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orderly  fashion.  For  instance,  if  you  know  that  a  stack  or  computer  folder  of  comments  came 
from  a  particular  site,  label  the  group  accordingly  and  keep  these  comments  separate  from 
comments  from  other  sites.  If  you  are  using  an  automated  survey,  you  have  the  advantage  of  not 
needing  to  transcribe  written  comments  to  a  computer  data  file  or  of  not  having  to  make  sense  of 
them  without  transcription.  You  will  use  these  comments  later  in  qualitative  analyses. 

If  you  used  an  optical-scan  approach,  scan  your  sheets  and  create  a  data  file  for 
quantitative  analyses.  If  you  used  automated  suryey  technology,  a  data  file  formatted  to  be 
compatible  with  standard  statistical  analysis  packages  may  be  created  automatically.  Keep  in 
mind  that  survey  disks  should  be  checked  for  viruses  when  then  are  returned.  If  you  used  a 
paper-and-pencil  survey  without  an  optical  scanning  capability,  enter  your  data  into  a  computer 
file  yourself.  When  you  create  your  data  file,  save  a  copy  on  a  separate  disk.  Make  the  format 
as  compatible  as  possible  with  the  data  analysis  software  you  intend  to  use. 

Select  Software  for  Data  Analysis:  There  are  many  computer-based,  commercial 
software  packages  available  to  help  you  analyze  your  results  and  provide  briefing  slides  or 
graphics.  For  instance,  there  is  SPSS  for  Windows  (Norusis,  1993a,  1993b)  or  EQS  from  BMDP 
(Bender,  1993a,  Bender  &  Wu,  1993).  Select  and  familiarize  yourself  with  the  package  you  wish 
to  use.  Ask  for  assistance,  if  necessary,  from  someone  who  knows  the  package  well. 

Sometimes,  analysis  programs  are  created  in-house  (by  computer  specialists  in  your  organization 
or  firm).  However,  unless  you  have  a  special  need,  it  is  usually  best  to  use  commercial-off-the- 
shelf  (COTS)  technology.  Don’t  only  be  concerned  about  the  ease  of  processing  your  data.  Also 
concern  yourself  with  how  well  your  software  produces  output  that  can  be  easily  understood  and 
interpreted.  For  instance,  can  it  array  the  data  that  you  want  to  compare  on  a  single  page  or  a 
few  pages,  or  do  you  have  to  use  several  different  reports  for  the  comparisons  that  are  relevant  to 
you?  You  do  not  want  to  waste  your  time  or  your  customer’s  time.  Therefore,  user-friendly 
output  is  critical.  You  may  also  want  to  provide  your  customer  with  the  capacity  to  analyze  the 
data  themselves  in  whatever  manner  they  want.  However,  if  you  do  this,  be  sure  that 
confidentiality  can  not  be  breached. 

Recode  and  Reverse  Score  if  Necessary:  Before  you  begin  your  analyses,  you  may 
need  to  recode  responses  if  your  response  scales  used  letters  since  you  need  numbers  to  perform 
computations.  Note  that  this  would  be  unnecessary  if  you  want  nothing  more  complex  than 
simple  frequencies  and  percents  of  people  responding  to  each  option  or  an  aggregate  response  to 
multiple  options  (  e.g.,  percent  “Agree”).  If  you  have  not  used  letters  or  numbers  (unnecessary 
on  an  automated  survey)  you  should  usually  assign  numbers  to  responses.  If  you  have  used 
negative  items,  you  will  probably  need  to  reverse  score  them.  This  is  done  to  make 
interpretation  easier  (for  instance,  so  that  high  scores  on  individual  items  signify  desirable 
outcomes).  It  is  also  done  so  items  can  be  properly  weighted  when  combined  with  the  responses 
to  positively  worded  items  for  calculating  multiple  item  scale  scores. 

If  your  responses  are  coded  as  letters  (alpha  coding)  convert  your  A,  B,  C,  D  etc.  to 
numbers  (numeric  coding)  such  as  1, 2, 3, 4,  etc.  Usually  the  more  of  something  you  want,  (like 
agreement  with  something  considered  good  or  positive)  the  higher  the  number.  Remember  also 
that  negatively  worded  items  are  usually  reverse  scored  (“Strongly  Disagree”  being  assigned  the 
highest  number)  since  agreement  with  such  an  item  is  not  positive. 

For  instance,  consider  the  following  response  scale  and  items: 
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Use  the  scale  below  to  indicate  your  agreement  or  disagreement  with  each  of  the  following  statements.  If 
you  don’t  know,  or  a  statement  is  not  applicable  to  you,  mark  response  “G.” 


A 

B 

C 

D 

E 

F 

G 

Strongly 

Disagree 

Disagree 

Slightly 

Disagree 

Slightly 

Agree 

Agree 

Strongly 

Agree 

Don’t  Know 

or 

Not  Applicable 

1 .  OMG  has  succeeded  in  establishing  integrated  teams  who  work  well  together. 

2.  The  OMG  structure  cannot  be  implemented  successfully  without  providing  additional  personnel. 

The  “A”  through  “F”  alpha  response  scale  would  be  recoded  numerically  as  “1”  through  “6”  or 
“6”  through  “1”  depending  on  the  positive  or  negative  wording  of  the  item.  For  item  1  in  the 
example  above,  the  coding  would  be  “1”  for  “Strongly  Disagree”  and  “6”  for  “Strongly  Agree.” 
For  item  2,  the  coding  would  be  “6”  for  “Strongly  Disagree”  and  “1”  for  Strongly  Agree.”  The 
“G”  escape  option  would  be  reported  separately  as  a  percent  selecting  this  option  and  excluded 
from  further  analysis.  Sometimes  it  is  treated  as  a  neutral  response  and  recoded  “3.5”  (for  the 
midpoint  of  this  6-point  scale)  in  analyses  involving  central  tendency  or  prediction.  However  it 
is  not  a  neutral  midpoint,  and  is  not  positioned  as  one  on  the  scale.  Thus,  this  technique  is 
discouraged. 

When  you  decide  to  reverse  score  or  not,  think  in  terms  of  “goodness”  or  desirability 
from  the  perspective  of  your  client.  If  you  do  decide  to  reverse  score,  high  scores  represent 
positive  (desired  or  good)  attitudes;  low  scores  represent  negative  attitudes  (which  are  not 
desired).  Note  that  having  negative  attitudes  is  not  necessarily  bad.  For  instance,  from  the 
perspective  of  the  respondent,  it  is  not  bad  to  have  a  negative  (realistic)  view  of  a  dysfunctional 
situation  or  to  negatively  evaluate  poor  leadership.  But  your  client  would  consider  a 
dysfunctional  situation  or  poor  leadership  to  be  undesirable. 

Sometimes  it  may  be  unnecessary  to  reverse  score.  Doing  so  or  not  depends  on  your  use 
of  the  information  and  how  it  is  presented.  When  you  do  reverse  score,  indicate  that  you  have 
done  so.  If  you  do  not  reverse  score,  explain  that  scores  will  have  different  meaning  depending 
on  the  wording  of  the  item. 

Construct  Multiple  Item  Scales:  When  you  perform  analyses,  some  of  them  will  be  at 
the  item  level.  However,  you  need  not  limit  yourself  to  item-by-item  analyses.  You  can 
combine  items  that  conceptually  or  empirically  belong  together  by  summing  the  scores  to  the 
items  and  dividing  by  the  number  of  items  (i.e.,  by  computing  the  “average”  or  “mean”  value  for 
the  cluster  of  items).  Then  you  can  compute  other  scale  score  descriptive  statistics  such  as  the 
mean,  standard  deviation  or  range. 

Scale  in  this  multi-item  form  is  defined  in  terms  of  multiple  items  being  combined  to 
form  a  scale,  rather  than  verbally  anchored  points  along  a  response  scale  continuum.  For 
instance,  you  may  have  a  construct  or  theme  called  “Communication”  measuring  the  free  flow  of 
communication  in  multiple  directions  and  across  multiple  organizational  units.  Say  it  is 
measured  by  four  items.  For  each  respondent,  you  would  sum  his  or  her  individual  responses  to 
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each  of  the  four  items  and  divide  by  four  to  produce  the  respondent’s  score  of  the  organization’s 
“Communication.”  If  he  or  she  left  an  item  or  items  composing  this  scale  blank,  you  can  either 
divide  by  only  the  number  of  items  in  the  scale  for  which  the  person  provided  responses  or 
substitute  the  midpoint  of  the  single  item  scale  for  each  missing  response  and  then  divide  by 
four. 


If  you  perform  inferential  analyses,  you  can  determine  if  your  original  (conceptual) 
selection  of  items  to  form  scales  was  justified  empirically  using  factor  and  internal  consistency 
analyses.  You  can  then  form  refined,  empirical  scales  based  on  this  information  and  use  them 
for  inferential  model  testing. 


Multiple  item  scales  provide  an  example  of  when  reverse  scoring  is  necessary.  It  would 
not  make  sense  to  include  positive  and  negative  items  in  the  computation  of  a  scale  score  if  you 
did  not  reverse  score  the  negative  ones  to  put  them  both  on  the  same  metric. 

Quantitative  Descriptive  Analyses  and  Results  Reporting 

Especially  in  management  consulting  or  program  evaluation  situations,  it  is  wise  to  keep 
your  analyses  and  results  reporting  simple.  Descriptive  statistics  such  as  means,  standard 
deviations,  frequencies  and  percents  can  be  computed  for  the  total  group.  (See  Norusis,  1993a, 
for  software  package  guidance.)  However,  you  will  probably  also  want  to  perform  analyses  and 
report  results  on  various  subgroups,  usually  based  on  background  (demographic)  data. 
Descriptive  statistics  may  be  all  you  need  in  most  situations.  If  you  do  compute  inferential 
statistics,  keep  your  reporting  of  these  statistics  simple  also.  Sometimes  demographic  subgroups 
are  aggregated  so  your  subgroups  are  not  too  small 

Why  should  analyses  and  results  reporting  typically  be  kept  simple  when  reporting  to  a 
non-academic  client?  Senior  leaders  are  busy  people  who  do  not  want  their  time  wasted.  They 
also  want  to  quickly  comprehend  information  provided  to  them  and  understand  the 
recommendations  that  follow  from  these  data.  Percents  are  easier  to  comprehend  than 
frequencies.  Thus,  simple  percents  graphically  presented  as  histograms  or  pie  charts  contrasting 
different  subgroups  may  suffice.  Aggregated  responses  to  a  single  item  can  even  be  presented. 
For  instance,  you  can  present  aggregate  agreement  (combining  the  percent  responding  to  those 
options  indicating  agreement— the  fourth,  fifth  and  sixth  options  on  the  author’s  6-point  scale). 
This  information  can  be  included  in  an  executive  summary  or  provided  to  your  client  in  a 
briefing  (a  formal  oral  presentation  with  slides). 

Information  on  percent  agreeing  or  percent  responding  to  each  response  category  should 
be  supplemented  with  additional  descriptive  statistics.  Means  and  standard  deviations  on 
individual  items  or  on  items  combined  to  form  scales  are  common  descriptive  statistics  which 
can  be  meaningful  to  your  client.  This  additional  descriptive  data  can  be  presented  in  briefings 
or  executive  summaries.  However,  too  much  of  this  kind  of  information  may  overwhelm  busy 
clients  who  are  not  used  to  analyzing  and  interpreting  such  data.  I  have  often  provided  clients 
with  notebooks  containing  more  detailed  descriptive  statistical  summaries.  Data  in  these 
notebooks  usually  provide  unaggregated  response  scale  data  (typically  item  frequencies, 
percents,  means  and  standard  deviations).  Keep  in  mind,  however,  that  data  should  remain 
aggregated  in  terms  of  individual  responses  being  combined  with  those  of  others  at  a  level  that 
prevents  disclosure  of  individual  identities.  Depending  on  the  sophistication  of  your  client,  such 
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a  notebook  may  also  contain  a  brief  tutorial  on  how  to  examine  and  interpret  this  information. 
For  instance,  these  simple  tutorials  can  explain  the  meaning  of  measures  of  central  tendency 
(usually  the  mean)  and  the  population  distribution  under  the  normal  curve.  They  should  also 
explain  the  meaning  of  a  standard  deviation.  For  instance,  you  could  state  that  about  68%  of  all 
responses  fall  within  plus  or  minus  one  standard  deviation  in  a  normal,  bell-shaped  distribution 
which  data  from  large,  well-selected  samples  usually  approximate.  Although  these  notebooks 
may  not  get  much  use,  this  information  should  be  made  available  for  customers  since  some  of 
them  may  want  more  detail. 

Do  Survey  Scales  Provide  Interval  Data?  I  was  recently  asked  if  survey  data  could  be 
considered  interval  data  and  if  it  is  legitimate  to  perform  statistical  analyses  on  such  data  using 
techniques  which,  at  least  in  theory,  require  interval  data.  He  responded  by  saying  that  the 
distances  along  the  Likert  scale  continuum  between  verbal  anchors  is  probably  not  in  equal 
intervals  although  it  is  usually  intended  to  be.  The  unequalness  of  the  intervals  would  be 
increased  if  the  verbal  anchors  marking  their  boundaries  are  poorly  worded.  However,  I  agree 
with  Nunnally  (1978,  p.  17)  who  argues  “...it  is  permissible  to  treat  most  of  the  measurement 
methods  in  psychology  and  other  behavioral  sciences  as  leading  to  interval  scales  (and  in  some 
instances  ratio  scales).”  He  goes  on  to  say  that  “...usually,  no  harm  is  done  in  most  studies  in  the 
behavioral  sciences  by  employing  methods  of  mathematical  and  statistical  analysis  which  take 
intervals  seriously.”  This  includes  measures  of  attitudes  and  personality.  The  implication  is  that 
most  statistical  analysis  methods  are  robust  to  violations  of  the  interval  data  assumption. 


Quantitative  Inferential  Analyses  and  Results  Reporting 

You  may  want  to  pursue  a  more  academic  approach  and  perform  inferential  analyses  in 
addition  to  descriptive  ones.  Even  if  you  compute  inferential  statistics,  keep  results  reporting 
simple  for  non-academic  audiences.  However,  if  reporting  in  a  journal  for  an  academic 
audience,  a  more  elaborate  presentation  would  be  expected.  This  may  include  graphic  depiction 
of  the  model  you  are  testing  and  coefficients  indicating  the  degree  of  association  or  prediction. 

In  inferential  analyses  you  determine  association,  for  instance,  by  computing  the 
correlation  between  items  or  clusters  of  items  (multiple  item  scales).  If  the  correlation  is  high, 
there  is  a  strong  association.  This  indicates  items  are  tapping  the  same  or  a  closely  related 
construct,  or  that  one  construct  (often  measured  by  multiple  item  scales)  is  related  to  another 
construct.  Inferential  statistics  are  also  used  for  prediction  and  model  testing  in  multivariate 
analyses  using  techniques  such  as  multiple  regression  or  structural  equation  modeling  and 
software  such  as  EQS  or  LISREL.  (See,  for  example,  Bentler,  1993;  Bentler  &  Wu,  1993; 
Cohen  &  Cohen,  1983;  Kerlinger  &  Pedhazur,  1973;  orNorusis,  1993b.)  Here  you  determine 
the  degree  to  which  various  independent  or  predictor  variables  “account  for  the  variance”  in  one 
or  more  criteria  (the  dependent  variable  or  variables).  By  accounting  for  a  substantial  amount  of 
this  variance,  which  can  be  attributed  to,  or  explained  by,  your  predictor  variables,  you  are 
demonstrating  the  validity  of  your  instrument  and  the  constructs  it  measures.  These  kinds  of 
analyses  are  used  to  test  research  hypotheses  or  determine  the  veracity  of  conceptual  models  of 
the  phenomena  being  studied.  Researchers  consider  themselves  successful  if  their  measured 
constructs  (which  define  an  implicit  or  explicit  model)  increase  the  explained  variance  in  a 
criterion  more  than  other  investigators  have  been  able  to  do.  This  kind  of  predictive  validity  is 
not  the  only  form  of  validity.  For  some  applications  even  face  validity  (consensus  that  the  items 
appear  to  measure  the  construct)  may  be  sufficient. 
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For  most  model  testing  applications,  multiple  items  are  combined  to  form  scales  or 
“operationally  defined  constructs.”  Earlier,  identifying  themes  and  writing  items  to  tap  them 
were  discussed.  The  items  you  write  to  measure  a  theme,  and  the  manner  in  which  you  combine 
them  to  form  a  scale,  constitutes  your  conceptual  operational  definition  of  a  construct.  However, 
without  further  assessment,  you  do  not  know  if  the  items  actually  hang  together  (measure  the 
same  underlying  construct)  empirically.  To  determine  this,  you  can  use  two  analysis  procedures. 
You  can  perform  factor  analysis  to  identify  items  which  cluster  together  empirically  (see,  for 
example.  Child,  1970  or  Gorsuch,  1983).  You  apply  labels  to  these  item  clusters  and  see  if  your 
conceptually  determined  constructs  (item  clusters  with  a  name  you  originally  chose)  are  the 
same  as  your  empirically  determined  constructs  (item  clusters  with  a  name  derived  from  factor 
analysis).  Of  course,  there  are  bound  to  be  differences.  However,  if  you  were  close,  you  did  a 
good  job.  You  can  also  compute  a  “coefficient  alpha”  to  determine  internal  consistency 
reliability,  the  most  commonly  used  index  of  reliability  (the  consistency  of  a  measure)  in  survey 
research.  With  this  technique  you  can  strengthen  your  empirical  constructs  by  determining 
which  items  to  delete  to  increase  alpha.  When  you  use  inferential  techniques  to  test  models  of 
societal  or  organizational  behavior,  you  use  your  empirically  derived  constructs  rather  than  your 
initial  conceptually  derived  ones.  You  can  also  compute  descriptive  statistics  such  as  means  for 
these  multi-item  constructs  by  summing  the  individual  item  scores  and  dividing  by  the  number 
of  items. 

If  you  are  reporting  results  of  inferential  analyses  for  an  academic  audience,  you  may 
want  to  provide  elaborate  statistical  summaries  of  coefficients  and  graphic  depictions  of  your 
conceptual  and  empirically  determined  models.  You  may  also  want  to  document  your  research 
in  a  journal  article  or  technical  report.  For  a  senior  leader  audience  in  the  Air  Force  or  a  business 
organization,  this  would  be  overkill.  When  reporting  to  an  executive  audience,  simplify  your 
presentation.  Avoid  academic  jargon.  Focus  on  the  implications  of  what  you  have  found  for 
management  practice  and  provide  clear,  practical  recommendations.  Report  your  findings  using 
multiple  modes  of  presentation:  informal  conversations  with  management,  executive  summaries 
and  formal  briefings. 


Qualitative  Content  Analyses  and  Results  Reporting 

You  will  likely  have  general  comments  pages  returned  from  many  of  your  survey 
participants.  These  will  contain  brief  statements  to  extended  narratives.  You  can  perform  a 
content  analysis  to  identify  themes  and  subthemes  in  these  data.  You  may  want  to  do  multiple 
content  analyses,  for  instance,  on  your  general  comments  data  and  on  item-specific  narratives  if 
you  provided  that  option  to  respondents.  This  process  is  not  sophisticated,  but  you  can  derive 
useful  information  from  it.  For  an  extended  description  of  content  analysis  see  Patton  (1980,  pp. 
299-326  and  Appendix  9.1). 

First,  identify  or  highlight  what  you  consider  to  be  interesting  or  important  content. 

Then  look  for  themes  and  subthemes  with  the  eventual  intent  of  creating  a  taxonomy  of  issues. 
As  your  theme  and  subtheme  taxonomy  begins  to  emerge  from  your  data,  write  down  an  outline 
of  themes,  like  a  table  of  contents.  Expect  this  outline  to  grow  and  change  as  you  gain 
understanding.  To  make  your  task  easier,  you  can  assign  numerical  codes  to  your  themes  and 
subthemes. 
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It  is  a  good  idea  to  input  your  narratives  into  a  computer.  This  will  allow  you  to  search 
for  key  words  or  to  move  your  narratives  around  as  you  begin  to  identify  themes.  However, 
before  doing  anything  that  might  alter  your  data  file,  keep  an  intact  copy  of  your  original 
comments  pages  and  transcriptions.  It  is  always  a  good  idea  to  save  “raw  data”  since  you  may 
want  to  start  from  scratch.  It  also  documents  your  starting  point  should  there  be  a  question  about 
your  data  later  or  should  some  other  researcher  want  to  perform  additional  analyses  (under  strict 
ethical  safeguards).  With  the  original  data  safely  stored  away,  you  can  then  use  scissors  or  your 
word  processor’s  cut  and  paste  capability  to  cull  out  unimportant  content  and  combine  narratives 
covering  particular  themes  or  subthemes. 


Computerized  Semantic  Analysis.  Computerized  semantic  analysis  procedures  could  be 
used  to  identify  common  themes  in  a  content  analysis  of  computerized  comments  or  notes.  Such 
a  procedure  matches  on  the  meaning  of  words  (nouns,  verbs,  etc.)  or  phrases.  The  procedure  has 
been  used  by  scientists  at  the  Armstrong  Laboratory  to  determine  the  similarity  of  tasks  when 
occupations  are  combined,  thus  requiring  the  merging  of  task  lists. 


People  may  criticize  content  analysis  as  unscientific  since  it  is  imprecise  and,  in  part,  an 
art  form.  The  themes  you  identify  may  not  be  the  themes  others  would  select  based  on  the  same 
input.  To  make  your  process  more  rigorous  and  less  subject  to  criticism,  have  several  people 
participate.  At  first  they  would  independently  identify  important  content  and  the  themes  and 
subthemes  it  contains.  After  each  person  goes  through  this  process,  collectively  discuss  the 
themes  identified  and  their  basis  for  them  and  begin  negotiating  and  consensus  building  to 
develop  a  taxonomy  everyone  can  agree  upon. 

When  you  report  the  results  of  your  content  analyses  to  your  client,  you  can  report 
themes  in  narrative  fashion  in  either  original  or  summarized  form.  You  can  also  count  the 
frequency  with  which  different  topics  are  mentioned  and  the  percent  of  people  mentioning  the 
themes.  When  reporting  to  your  customer,  you  can  even  quote  some  of  the  most  representative 
comments  on  critical  themes,  even  if  you  originally  summarized  this  information.  When  you 
summarize  what  others  have  said,  you  risk  interjecting  your  own  meaning.  Thus,  it  is  a  good 
idea  to  supplement  summaries  with  representative  quotations. 


Additional  Analyses 

Cross-validation:  Sometimes,  to  test  the  generalizability  of  results  across  samples, 
cross-validation  is  used.  In  this  procedure,  weights  (for  instance,  from  a  regression  analysis) 
from  one  sample  are  applied  to  another  sample.  The  other  sample  can  even  be  a  subset  of  your 
whole  sample.  The  objective  is  to  see  if  the  variance  accounted  for  by  your  predictors  remains 
about  the  same.  This  procedure  would  not  need  to  be  used  in  most  survey  work  for  management 
consultation  purposes. 

Bias  Analyses:  You  may  want  to  determine  if  the  responses  you  received  appear  to  be 
representative  of  your  target  population;  that  is,  they  are  not  biased  because  they  were  obtained 
from  an  unrepresentative  subset  of  it.  If  you  know  the  identity  of  respondents  and 
nonrespondents,  you  can  determine  the  demographic  profile  of  each  of  these  groups  and  compare 
them.  Demographic  profile,  as  used  here,  simply  means  the  percent  of  people  in  each 
demographic  subcategory.  Are  they  similar?  If  not,  you  may  have  bias.  Usually  you  do  not 


73 


know  precisely  who  did  not  respond.  In  this  case  you  can  compare  those  who  did  respond  with 
the  total  target  population  and  see  if  their  demographic  profiles  are  similar. 

An  automated  survey  can  help  reduce  bias  because  it  can  facilitate  identification  of 
people  who  did  not  respond  and  distribution  of  a  follow-up  message  or  follow-up  survey.  Since 
an  automated  survey  can  easily  track  response  or  non-response,  it  can  aid  in  bias  analyses. 


Multiple  Time  Assessment 

Data  analysis  of  either  a  qualitative  or  quantitative  nature  is  not  necessarily  a  one-time  event. 
You  may  want  to  assess  changes  in  attitudes  or  behaviors  over  time.  You  may  be  interested  in 
assessing  improvements  in  performance  or  in  acceptance  of  a  product  or  service.  You  may  be 
evaluating  the  effectiveness  of  a  program,  product  or  service  before  and  after  you  intervene  to 
improve  it.  This  is  sometimes  referred  to  as  “quasi-experimental  designs”  using  Time  1-  Time  2 
assessments  (Campbell  &  Stanley  1963,  Cook  &  Campbell,  1979)  and  it  is  an  approach  often 
used  in  program  evaluation  and  marketing  research.  Sometimes  programs  or  products  are 
assessed  while  they  are  undergoing  improvement.  At  other  times  they  are  evaluated  after  they 
have  been  refined  as  much  as  they  are  going  to  be.  This  is  the  distinction  between  formative  and 
summative  evaluation.  Also,  in  any  kind  of  multiple  time  assessment,  researchers  are  not  just 
interested  in  the  results  at  any  given  time,  but  in  comparing  and  contrasting  results  across  time, 
looking  for  improvements  or  trends. 

A  tendency  people  new  to  surveys  have  is  to  misinterpret  any  change  across  time  as  an 
important  change.  For  instance,  I  was  once  asked  to  help  interpret  trends  from  the  results  of  two 
major  command  quality  surveys  taken  about  a  year  apart.  Command  personnel  thought  they  had 
identified  significant  changes,  some  of  which  were  disturbing.  The  changes  across  time  were  so 
minor  that  I  advised  command  personnel  to  interpret  the  results  as  natural  statistical  variation 
that  was  of  no  practical  consequence.  This  interpretation  might  also  apply  to  small  differences  in 
characteristics  of  large  samples  which  may  be  satistically  significant  but  of  little  practical 
significance. 
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Postscript:  By-Chapter  Summary  of  Recommendations 
Data  gathering  Method  (Chapter  2) 

Don’t  automatically  select  a  paper-and-pencil  survey.  Consider  other  forms  of  surveys  like  an 
automated  survey  or  a  telephone  interview.  Even  consider  other  methods  such  as  a  face-to-face 
interview.  Select  the  method  most  suited  to  your  needs. 

Motivating  Participation  (Chapter  3) 

Take  steps  to  motivate  participation  to  increase  response  rate.  Explaining  the  importance  of  the 
survey  in  a  cover  letter  or  in  introductory  comments  is  particularly  important,  as  is  emphasizing 
the  voluntary  nature  of  the  survey  and  your  commitment  to  confidentiality. 

Survey  Ethics  (Chapter  4) 

Since  you  create  the  conditions  for  unconstrained  speech  and  may  gather  information  about 
potentially  offensive  attitudes  or  behaviors,  or  even  illegal  actions,  you  must  always  protect  your 
respondents  from  harm. 

Ensure  that  participation  is  voluntary  and  don’t  use  coercion  to  get  people  to  respond  to  your 
survey  or  to  any  part  of  it. 

Maintain  confidentiality.  Never  share  an  individual  respondent’s  data  with  anyone  without  his 
or  her  informed  consent,  except  for  comments  data  which  may  be  transformed  verbatim  (with 
references  to  specific  individuals  removed).  Only  share  aggregated  information  with 
management  (or  other  clients).  Your  aggregation  should  be  sufficiently  large  that  specific 
individuals  would  not  be  identified  by  their  demographic  characteristics.  Prevent  unauthorized 
access  to  your  raw  data. 

If  your  ethical  responsibilities  place  you  in  conflict  with  organizational  or  societal  expectations, 
explain  the  basis  of  your  ethical  position  and  try  to  resolve  the  issue  in  a  way  that  is  consistent 
with  professional  ethical  standards. 

Limit  content  that  is  very  objectionable  or  intrusive,  if  possible.  If  you  must  include  such 
content,  be  sensitive  to  how  such  items  are  worded,  emphasize  confidentiality,  and  always  allow 
respondents  to  not  respond  without  the  threat  of  adverse  consequences.  Substitute  specific 
questions  pertaining  to  the  respondents  themselves  with  more  general  attitudinal  items  pertaining 
to  the  objectionable  issues. 

In  surveys  used  for  evaluation  purposes  which  could  result  in  the  denial  of  an  opportunity  (for 
employment,  promotion,  etc.),  avoid  asking  for  information  which  could  be  used  to  discriminate 
against  individuals  on  the  basis  of  some  unfair  personal  characteristic  such  as  age,  ethnicity, 
race,  gender,  marital  status,  or  religion.  Such  items  are  often  illegal. 

Avoid  asking  questions  regarding  specific  job-irrelevant  organizational  affiliations  since  legal 
precedent  protects  organizations  and  their  members  from  the  forced  disclosure  of  such 
information  since  it  could  be  used  to  harm  the  members.  Substitute  more  general  attitudinal 
items  pertaining  to  the  organizations  of  interest  or  their  members. 
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If  the  survey  developer  does  not  follow  the  ethical  guidelines  discussed,  prospective  survey 
participants  have  the  right  to  refuse  to  complete  the  survey.  Alternatively,  they  could  skip  items 
they  consider  objectionable  or  too  intrusive. 

Sampling  (Chapters) 

Make  your  sample  as  representative  of  your  target  population  as  possible,  either  through  random 
sampling  or  purposeful  sampling.  If  necessary,  oversample  underrepresented  groups. 

Ensure  that  your  sample  size  is  sufficiently  large  to  make  up  for  nonresponse  and  to  allow  you  to 
have  confidence  in  total  sample  and  subgroup  analyses. 

Survey  Format  Decisions  (Chapter  6) 

Select  the  most  appropriate  t5^e  of  survey  for  your  own  needs.  If  you  choose  a  paper-and- 
pencil  survey,  decide  if  you  want  an  optical  scanning  capability. 

Write  your  survey  to  the  reading  level  of  your  intended  respondents.  Eliminate  any  language 
barriers,  if  necessary,  by  creating  foreign  language  versions  or  by  providing  translators. 

Select  the  background  or  demographic  information  you  will  need  to  perform  meaningful 
subgroup  analyses. 

Divide  your  survey  into  meaningful  parts.  Do  not  cluster  all  items  measuring  the  same  construct 
together. 

Item  Construction  (Chapter  7) 

Decide  what  kinds  of  items  you  wish  to  use:  items  with  narrative  responses,  items  with  item- 
specific  responses,  or  items  with  common  response  scales.  Use  a  mixture  if  you  wish.  Use 
items  with  common  response  options  as  much  as  possible  to  make  the  survey  easier  to  take. 

A  Likert  scale  format  is  recommended,  although  other  scale  forms,  such  as  the  semantic 
differential,  are  also  acceptable. 

For  items  using  common  scales,  a  6-point  scale  with  an  escape  option  and  no  neutral  midpoint  is 
recommended. 

Providing  verbal  anchors  at  multiple  points  along  a  scale  continuum  is  recommended  rather  than 
just  providing  anchors  at  each  end. 

Be  open  to  improvements  in  scaling  techniques  that  are  made  possible  with  computers.  For 
instance,  continuous  scales  where  respondents  can  select  any  point  along  a  continuum  will 
probably  replace  categorical  scales  in  the  future. 

Research  needs  to  be  conducted  on  improving  scaling  techniques. 
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An  ^‘escape  option”  is  recommended  in  most  instances  to  allow  respondents  to  indicate  that  an 
item  is  not  applicable,  or  that  he  or  she  does  not  know  how  to  respond  to  it.  This  option,  which 
is  not  a  part  of  the  scale,  should  be  visually  apart  from  the  scale  continuum.  Thus,  if  a  graphic 
scale  is  used,  the  line  representing  the  response  continuum  should  not  extend  to  the  escape 
option. 

Assigning  numerical  values  to  the  escape  option  as  if  it  were  a  neutral  midpoint  response  for 
analysis  purposes  is  not  recommended.  For  example,  do  not  transform  responses  to  the  escape 
option  into  a  score  of  3.5  on  a  6-point  scale. 

Keep  your  items  short  and  as  simple  as  possible. 

Make  sure  your  item  responses  are  mutually  exclusive  (non-overlapping)  and  exhaustive  (they 
cover  every  possible  option,  even  if  just  with  “Don’t  Know  or  Not  Applicable,”  “Other”  or 
“More  than”  or  “Less  than”  response  categories). 

Don’t  mix  apples  and  oranges;  that  is,  each  item  should  cover  one  topic  rather  than  multiple 
topics  or  facets  of  a  topic. 

Don’t  be  too  vague  in  how  you  write  your  item  and  try  to  tap  directionality.  For  instance,  if  you 
ask  an  item  about  leadership,  be  specific  about  how  leadership  led  to  a  valued  outcome.  Also, 
try  to  determine  if  your  respondent  wanted  more  of  something  or  less  of  it,  so  you  will  know 
what  action  to  take  to  address  his  or  her  concern.  For  instance,  does  a  person  want  more 
responsibility  or  less  responsibility?  Do  not  assume  you  know  the  direction  desired  or  you  may 
be  imposing  your  value  system  on  the  respondent. 

Include  negative  items  in  your  survey  to  break  a  tendency  toward  an  “acquiescent  response  set” 
and  to  let  the  respondent  know  you  are  in  touch  with  the  issues  and  are  not  trying  to  put  a 
positive  spin  on  the  results. 

Include  items  only  on  topics  your  client  can  do  something  about.  Otherwise  you  waste  the 
respondent's  time  and  may  unrealistically  increase  expectations. 

Keep  your  items  relevant  to  your  purpose.  Otherwise  you  are  wasting  the  respondent’s  time. 

This  is  especially  true  if  the  items  are  overly  intrusive.  If  you  ask  an  intrusive  question  you  have 
no  need  to  know,  you  go  beyond  just  wasting  time  to  inappropriately  invading  privacy. 

If  you  must  include  objectionable  or  intrusive  items,  phrase  them  in  a  way  that  will  be  more 
acceptable  to  respondents.  For  example,  instead  of  asking  respondents  about  their  specific 
behaviors  or  attitudes,  phrase  your  items  in  terms  of  their  attitudes  toward  people  in  general  who 
behave  in  that  way  or  who  hold  certain  views. 

Comments  (Chapter  8) 

Always  include  a  general  comments  section  of  at  least  one  page  at  the  end  of  a  survey.  This  will 
allow  people  to  communicate  with  you  in  their  words. 
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If  possible,  also  provide  the  opportunity  for  item-specific  comments.  These  can  be  short 
narratives.  This  is  quite  feasible  on  an  automated  survey  or  in  a  computer  assisted  telephone 
interview. 

Provide  a  thank  you  message  and  a  return  reminder  at  the  end  of  the  survey. 

Creating  a  Survey  (Chapter  9) 

Identify  in  advance  the  themes  you  wish  to  measure.  Don’t  just  start  writing  items. 

Consult  many  sources  to  gather  information  about  what  to  ask:  the  literature,  scholars,  your 
clients,  and  representatives  of  the  target  population. 

See  if  there  are  existing  items  you  might  use  before  you  begin  writing  your  own  items. 

At  first,  just  generate  as  many  item  ideas  as  you  can  without  judging  their  quality.  Then  cull  out 
the  bad  ones  and  refine  those  with  promise. 

Put  together  a  draft  survey,  realizing  that  you  may  need  to  go  through  several  iterations.  Enlist 
the  help  of  your  colleagues  and  pretest  your  draft.  Interview  pretest  participants.  On  the  basis  of 
peer  review  and  pretest  feedback,  finalize  your  survey. 

Consider  the  need  to  get  your  survey  approved.  If  you  are  in  the  Air  Force,  you  may  need  to  get 
a  survey  control  number  from  HQ  AFPC.  If  you  are  a  Government  employee  and  are  surveying 
non-govemment  civilians,  you  may  need  0MB  clearance.  You  may  also  need  to  get  an 
exemption  from  an  institutional  review  board  concerned  with  the  ethical  use  of  human  subjects. 

Data  Analysis  (Chapter  10) 

If  you  used  an  optically  scanned  survey  or  separate  scan  sheet,  make  sure  it  is  in  good  condition 
for  scanning.  Correct  problems.  For  example,  if  the  sheets  or  booklets  are  not  filled  out 
correctly,  reaccomplish  this  task  yourself.  Make  sure  you  have  recorded  the  person’s  responses 
faithfully. 

Perform  a  veracity  check  to  determine  if  respondents  were  conscientious  when  filling  out  the 
survey.  If  this  was  obviously  not  the  case  for  some,  eliminate  these  respondents  from  your 
sample. 

Create  a  data  file  compatible  with  the  analysis  software  you  intend  to  use. 

Shift  from  alphabetic  to  numeric  coding,  and  reverse  score,  if  necessary  (for  instance,  in  most 
instances  where  you  have  negative  items). 

If  you  want  to  perform  inferential  analyses,  construct  multiple-item  scales.  Empirically 
determine  the  adequacy  of  your  conceptual  scales  using  factor  analysis  and  by  computing 
internal  consistency  reliability  (coefficient  alpha). 

As  a  minimum,  perform  quantitative  descriptive  analyses  and  prepare  your  results  for  reporting 
to  clients  and  others.  Keep  your  descriptive  analyses  and  reporting  simple. 
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If  you  desire  more  advanced  analyses,  perform  quantitative  inferential  analyses.  These  analyses 
will  be  more  complex,  but  may  be  better  suited  for  an  academic  audience. 

Perform  a  qualitative  content  analysis  of  the  data  you  have  collected  from  your  comments  pages. 

If  desired,  determine  the  stability  and  generalizability  of  your  findings  by  performing  a  cross 
validation. 

If  you  are  concerned  about  response  bias,  perform  bias  analyses.  One  simple  approach  is  to 
determine  if  the  demographic  characteristics  of  those  who  responded  approximate  the 
demographic  characteristics  of  the  target  population. 
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Appendix  A:  Cover  Letters  From  the  OMG  Survey  and  the  FM  Career  Survey 
Dear  Air  Force  Medical  Service  Member 

Last  year  we  implemented  a  new  structure  in  Air  Force  Medical  Treatment  Facilities 
(MTF’s)  called  the  Objective  Medical  Group  (OMG).  Changes  brought  about  by  this  new 
structure  are  far  reaching  and  may  have  had  an  effect  on  the  quality  of  your  worklife  and  the 
quality  of  medical  care  your  MTF  provides.  You  may  view  some  of  these  changes  and  their 
effects  as  positive  while  considering  others  to  be  negative. 

In  my  role  as  your  Surgeon  General,  I  am  personally  committed  to  providing  you  with  a 
work  environment  that  1)  satisfies  your  needs,  2)  facilitates  rather  than  hinders  your  doing  your 
job  to  the  best  of  your  ability,  and  3)  provides  patients  with  quality  care  at  reasonable  cost.  To 
accomplish  these  goals,  I  need  to  know  your  views  about  the  OMG  structure  and  its 
implementation  to  date.  By  responding  frankly  to  this  survey,  you  will  give  me  and  other  senior 
medical  service  leaders  the  understanding  we  need  to  make  improvements. 

Although  the  OMG  survey  is  voluntary,  I  strongly  urge  you  to  take  it.  You  will 
probably  be  able  to  complete  this  automated  survey  in  about  thirty  minutes.  You  need  not  worry 
about  privacy  since  the  survey  is  also  confidential.  We  do  not  ask  for  your  name  or  other 
personal  identifying  information  except  for  general  demographic  data  needed  for  analysis 
purposes.  You  will  be  able  to  comment,  briefly  on  any  item,  and  more  extensively  in  the 
comments  section  at  the  end  of  the  survey.  Except  for  your  comments,  which  will  be  transcribed 
verbatim  (without  personal  identifying  information),  your  individual  responses  will  not  be  shared 
with  me  or  with  others  in  your  chain  of  command.  They  will  be  combined  with  those  of  others 
for  analysis  purposes. 

Please  take  the  time  to  complete  this  survey.  I  hope  you  find  the  automated  format 
easier  and  faster  to  use  than  a  paper-and-pencil  survey.  I  will  provide  you  with  a  summary  of 
results  and  feedback  on  how  we  plan  to  use  this  information  to  bring  about  needed 
improvements.  Thanks  for  helping  to  make  our  MTFs  better  places  in  which  to  work  and  receive 
good  medical  care. 


Sincerely 


EDGAR  R.  ANDERSON,  JR. 
Lieutenant  General,  USAF,  MC 
Surgeon  General 
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From  the  Desk  of  Mr.  Hale 


Dear  Air  Force  Financial  Management  Personnel: 

This  survey  solicits  your  views  about  our  financial  management  career  field.  We  are 
interested  in  your  thoughts  about  FM  career  management,  the  adequacy  of  career  information,  and  the 
availability  of  senior  leaders  to  answer  career  questions.  We  also  want  to  know  how  you  view  future 
career  prospects  in  Air  Force  financial  management.  This  survey  looks  at  the  FM  career  field  in  general 
and  is  not  a  Financial  Management  Career  Program  evaluation. 

What  will  we  do  with  the  survey  results?  We  will  use  them  as  a  guide  when  assessing  and 
revising  our  training  programs,  mentoring  efforts,  and  career  practices.  If  the  results  indicate  a  need,  we 
will  develop  new  programs  and  practices.  So  that  you  know  what  the  results  told  us,  we  will  provide  you 
a  summary  of  survey  results  in  our  Comptroller  magazine  and  on  our  new  Home  Page  on  the  World 
Wide  Web. 

Although  it  is  voluntary,  I  strongly  urge  you  to  complete  this  survey.  I  hope  that  the  new 
automated  format  makes  it  easier  to  take  and  I  expect  that  most  people  will  complete  the  survey  in  about 
30  minutes  or  less.  You  need  not  worry  about  privacy  since  the  survey  is  confidential.  We  do  not  ask  for 
your  name  or  other  personal  identifying  information  except  for  some  general  demographic  data  that  we 
need  to  analyze  the  results.  Except  for  any  comments,  which  will  be  transcribed  verbatim  without  personal 
identifying  information,  your  individual  responses  will  not  be  shared  with  me  or  with  others  in  your  chain 
of  command. 

Thanks  for  your  help. 


ROBERT  F.  HALE 

Assistant  Secretary  of  the  Air  Force 

Financial  Management  and  Comptroller 


Appendix  B:  Introductory  Comments  and  Instructions 
from  the  San  Antonio  Air  Logistics  Center  Career  Planning  Survey 


The  recent  changes  brought  about  by  the  BRAC  Realignment  and  Closure  List  are  far- 
reaching  and  extend  to  all  of  you  and  your  families.  This  is  a  difficult  time  for  you  as  you  begin 
preparing  for  your  future  careers,  or  for  retirement. 

To  assist  you  in  planning  for  your  future,  the  senior  leadership  at  the  SA-ALC  needs  to 
know  your  views  about  the  BRAC  decision  and  how  it  may  affect  you.  By  responding  frankly  to 
this  survey,  you  will  give  senior  staff  members  vital  information  that  we  can  use  to  help  you  and 
your  fellow  workers  make  a  successful  transition. 

Please  take  the  time  to  complete  this  survey.  We  will  provide  you  with  a  summary  of 
the  results  and  feedback  on  how  we  plan  to  use  this  information.  Thanks  for  helping  by  sharing 
your  concerns  and  opinions. 


General  Instructions: 

For  all  but  Part  3 :  Comments,  use  the  general  purpose  answer  sheet  provided  to  respond  to  this 
survey. 

Mark  your  responses  with  a  number  2  pencil.  Ensure  that  your  marks  are  heavy  and  fully 
blacken  the  oval.  Avoid  stray  marks  or  marks  that  go  outside  the  ovals. 

Note:  You  may  want  to  include  examples  of  the  right  and  wrong  way  to  mark  the  answer 
sheet  at  this  point. 

This  survey  is  anonymous,  confidential  and  voluntary.  Except  for  possible  verbatim 
transcription  of  comments  (from  which  personal  identifying  information  will  be  removed)  only 
grouped  data  will  be  shared  with  management. 

Make  any  comments  in  Part  3,  located  on  the  last  pages  of  the  survey. 

Do  not  staple  or  fold  the  answer  sheet  because  the  scanner  may  not  read  the  form  and  your 
responses  would  not  be  scanned. 

Once  you  have  completed  this  survey,  return  your  answer  sheet  and  comments  pages  to  your 
local  survey  administrator  or  by  mail  directly  to  the  Human  Resources  Directorate  of  the 
Armstrong  Laboratory  at  the  following  address: 

AL/HRMJ 

7909  Lindbergh  Drive 
Brooks  AFB,  TX  78235-5352 
Attn.:  Dr.  Tom  Watson 

If  you  have  questions,  or  comments  contact  your  local  survey  administrator. 
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Appendix  C:  Rights  Perspective:  Free  Speech  and  Privacy 

Overview:  Americans  have  the  right  to  speak  freely  and  surveys  can  facilitate  this. 
However,  unconstrained  speech  in  response  to  a  survey  is  unlikely  unless  steps  are  taken  to 
protect  individual  privacy.  This  is  achieved  by  not  forcing  participation,  by  making  surveys 
anonymous  (or  keeping  the  responses  of  individuals  confidential),  and,  with  few  exceptions,  by 
sharing  only  aggregated  data.  Survey  developers  must  do  these  things  not  only  to  get  accurate 
data,  but  also  to  honor  their  ethical  obligations  to  respondents.  If  privacy  is  guaranteed, 
respondents  are  protected  from  harm  and  their  dignity  is  maintained.  Thus,  their  motivation  to 
participate  and  to  provide  undistorted  information  is  likely  to  be  increased. 

The  ethical  justification  for  voluntary  participation  and  confidentiality  is  based  on  two 
basic  human  rights:  free  speech  and  privacy,  as  discussed  below. 

Guaranteeing  Free  Speech:  Some  of  our  founding  fathers  considered  the  Constitution, 
as  originally  written  in  1787,  flawed.  They  believed  it  did  not  sufficiently  protect  individuals 
against  the  power  of  government.  After  years  of  debate,  the  first  ten  amendments  (the  Bill  of 
Rights)  were  added  to  our  Constitution  in  1791  to  protect  individual  liberties  (Glasser  & 
Adelman,  1991;  Alderman  &  Kennedy,  1991).  Alderman  and  Kennedy  (1991,  pp.  13  <&  15) 
explain  that  these  amendments,  collectively,  “outline  the  most  comprehensive  protection  of 
individual  freedom  ever  written;”  however,  in  a  1987  poll,  59  percent  of  Americans  queried 
could  not  identify  this  document.  It  is  not  that  these  amendments  give  us  rights.  As  Glasser  and 
Adelman  (1991)  indicate,  our  founders  believed  such  rights  as  free  speech  were  fundamental 
rights  of  human  beings.  The  Bill  of  Rights  was  added  to  protect  such  inalienable  rights. 

The  First  Amendment  protects  religious  freedom,  freedom  of  speech  (and  expression), 
freedom  of  the  press,  the  right  of  people  to  peacefully  assemble,  and  the  right  to  petition  the 
government  for  a  redress  of  grievances.  (See  Text  of  the  Constitution  of  the  United  States,  1992, 
for  the  specific  wording  of  this  and  other  amendments.  For  a  discussion  and  description  of  legal 
cases  pertaining  to  each  amendment,  see  Alderman  &  Kennedy,  1991 .)  It  took  a  long  time  for 
First  Amendment  rights  to  be  realized  in  practice  (Glasser  &  Adelman,  1991,  Chapter  3). 
However,  this  amendment  helped  protect  the  right  of  Americans  to  speak  and  otherwise  express 
themselves  in  largely  unconstrained  ways  and  to  peacefully  protest  government  actions. 

In  exploring  the  historical  defense  of  free  speech,  Alderman  and  Kennedy  (1991,  p.  32) 
cite  the  views  of  John  Milton,  and  more  recently.  Justices  Holmes  and  Brandeis.  The  basic  idea 
is  that  people  should  be  free  to  discuss  and  debate  ideas  without  the  censorship  of  prevailing 
doctrine.  To  paraphrase  Homes  and  Brandeis,  in  the  free  market  place  of  ideas,  good  counsels 
remedy  bad  ones.  For  free  speech  advocates,  the  antidote  for  offensive  speech  in  a  free, 
pluralistic  society  is  counterspeech,  not  prohibition.  As  Alderman  and  Kennedy  explain,  this 
view  is  based  on  idealism  and  respect  for  the  principle  of  free  speech,  even  if  you  dislike  the 
unreasonable  or  intolerant  views  that  free  speech  may  unleash.  Ultimately,  it  was  thought,  good 
ideas  and  truth  would  prevail. 

The  First  Amendment  protection  of  a  free  press  also  supports  free  speech.  In  a  1964 
deliberation  over  First  Amendment  protections  of  a  free  press,  the  Supreme  Court  indicated  a 
“profound  national  commitment  to  the  principle  that  debate  on  public  issues  should  be 
uninhibited,  robust  and  wide-open”  (Alderman  &  Kennedy,  1991,  p.  46). 
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Permitting  free  speech  does  not  guarantee  it  will  occur.  Despite  constitutional 
safeguards,  speech  and  other  forms  of  expression  are  usually  constrained.  Some  laws  constrain 
speech  and  expression.  More  pervasively,  organizational,  family,  group  and  societal  norms 
constrain  speech  to  that  which  is  accepted  or  expected  in  certain  contexts.  Fear  of  harassment  or 
punishment  also  constrains  speech.  Thus,  much  goes  unsaid.  Much  of  what  remains  unsaid 
could  be  helpful  to  parents,  teachers,  health  and  social  service  agencies,  politicians, 
manufacturers,  employers,  and  others  who  wish  to  understand  and  effectively  deal  with  a  wide 
array  of  organizational,  social,  and  consumer  issues.  For  instance,  consider  these  situations; 

•  An  Air  Force  commander  or  a  corporate  chief  executive  officer  wants  to  know  what 
personnel  practices  are  disliked  by  employees  and  their  suggestions  for  making  them  better. 

•  Parents,  teachers,  and  school  and  community  leaders  want  to  know  about  the  nature  of  gang 
activities  among  middle  and  high  school  students  in  their  district. 

•  Politicians  and  law  enforcement  officials  want  to  know  about  attitudes  toward  and  use  of 
illicit  drugs  in  their  community. 

•  Politicians  want  to  know  the  attitudes  of  citizens  toward  the  economy,  crime,  welfare  and  the 
environment,  and  changes  in  these  attitudes  over  time. 

•  A  manufacturer  with  slumping  sales  wants  to  know  how  to  produce  products  customers  will 
really  want  and  be  more  likely  to  buy. 

•  Senior  military  leaders  want  to  know  the  support  they  have  for  a  planned  deployment  a  few 
days  away. 

In  some  of  these  situations,  those  in  the  best  position  to  provide  information  would  be 
unlikely  to  volunteer  it  and  they  would  have  limited  access  to  decision  makers.  In  addition, 
without  a  systematic  way  to  collect  information  from  those  who  could  provide  it,  the  input 
received  may  be  biased.  It  may  come  mostly  from  those  discontented  enough  to  contact  the 
officials— or  from  their  immediate  colleagues.  In  the  last  situation  described,  decision  makers 
may  not  get  accurate  information  rapidly  enough. 

Protecting  Privacy:  People  have  as  much  right  not  to  speak  as  to  speak.  Our  founding 
fathers  were  sensitive  to  the  right  of  persons  not  to  express  themselves  if  they  wished—to  be  let 
alone,  having  a  right  to  privacy.  They  had  experienced  first-hand  oppression  from  a  distant 
government  and  many  early  settlers  came  to  America  to  escape  religious  persecution.  Influenced 
by  John  Locke,  they  also  believed  in  natural  or  fundamental  rights,  independent  of  government 
and  limiting  its  power  (Glasser  &  Adelman,  1991;  The  Constitution  of  the  United  States:  A 
Commentary,  1982). 

Despite  the  sensitivity  of  our  founding  fathers,  the  Bill  of  Rights  is  vague  about  privacy. 
As  Shattuck  (1977,  p.  49)  notes,  “The  right  of  ‘privacy  of  association’  and  the  ‘right  to 
anonymity’  are  nowhere  explicitly  delineated  in  the  constitutional  text.”  However,  there  are 
several  amendments  which  involve  privacy  interests  (see  Shattuck,  1977,  Kauper,  1990).  The 
courts  have  established  that  the  First  Amendment  includes  privacy  linked  to  the  freedom  of 
association  within  its  “penumbra,”  that  is,  in  a  peripheral,  less  clearly  defined  region.  The  Third 
Amendment  prevents  peacetime  quartering  of  soldiers  in  homes  without  the  owners  consent,  nor 
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in  times  of  war,  except  as  prescribed  by  law.  The  Fourth  Amendment  ensures  that  people  must 
be  “secure  in  their  persons,  houses,  papers  and  effects  against  unreasonable  searches  and 
seizures.”  The  Fifth  Amendment,  protecting  people  from  self-incrimination,  proclaims  that  no 
person  shall  be  “a  witness  against  himself.”  The  Fifth  and  Fourteenth  amendments  also  protect 
persons  from  the  deprivation  of  life,  liberty  or  property  without  due  process  of  law.  The 
Fourteenth  Amendment  also  prevents  the  states  from  denying  constitutionally  protected  rights. 

In  recent  times,  the  1965  Griswold  v.  Connecticut  case  is  famous  for  helping  to  establish 
the  right  to  privacy.  Justice  Douglas,  in  a  decision  involving  this  case  (see  Beaney,  1990,  p.4), 
concluded  that  earlier  cases  “suggested  that  specific  guarantees  in  the  Bill  of  Rights  have 
penumbras  formed  by  emanations  from  those  guarantees  that  help  give  them  life  and  substance.” 
According  to  Beaney,  Douglas  then  asserted  that  the  First,  Third,  Fourth,  Fifth  and  Ninth 
amendments  provide  “zones  of  privacy.”  As  Kaukper  (1990)  discusses,  Douglas  spoke  of  rights 
that  were  peripheral  (hence  the  “penumbra”  terminology)  to  those  explicitly  expressed  in  the 
first  eight  amendments,  but  without  them,  the  more  explicit  ones  would  be  less  secure. 

The  Ninth  Amendment,  is  sometimes  cited  as  the  most  compelling  Constitutional 
amendment  dealing  with  privacy  rights  because  it  is  generic,  pertaining  to  retained, 
unenumerated  rights  that  should  not  be  denied.  Alderman  and  Kennedy  (1991)  discuss  the  Ninth 
Amendment  almost  exclusively  in  terms  of  privacy  protection,  and  Shattuck  (1977,  p.  xiv), 
states: 


“to  the  extent  that  historic  or  novel  privacy  interests  may  not  be  sufficiently  protected  by 
the  First,  Fourth,  Fifth,  and  Fourteenth  Amendments,  it  has  sometimes  been  suggested 
that  they  are  safeguarded  by  the  Ninth  Amendment,  which  provides  that  ‘the 
enumeration  in  the  Constitution,  of  certain  rights,  shall  not  be  construed  to  deny  or 
disparage  those  retained  by  the  people.” 

There  is  debate  over  the  applicability  of  this  amendment  to  privacy.  However,  in 
Griswold  v.  Connecticut,  the  majority  of  US  Supreme  Court  Justices  invoked  the  ninth 
amendment  to  support  the  right  to  marital  privacy  (Emerson,  1990).  In  more  recent  Supreme 
Court  decisions,  substantive  due  process  has  been  used  to  defend  privacy  rights  (see  Alderman  & 
Kennedy,  1991,  pp.  321-323  &  pp.  387-388).  This  concept  refers  to  fundamental, 
nonprocedural,  individual  rights  not  explicitly  articulated  in  the  Bill  of  Rights  but  considered 
indispensable  in  a  free  society. 

Although  privacy  interests  were  only  partly  articulated  in  the  Bill  of  Rights  and  the  legal 
basis  for  the  right  to  privacy  was  clarified  only  recently  in  court  decisions,  this  concept  has  a 
long  history  in  American  common  law.  In  1 890,  an  article  appeared  in  the  Harvard  Law  Review 
that  created  a  new  chapter  (Hixon,  1987)  or  revolution  (Shattuck,  1977)  in  common  law.  The 
article  titled  “The  Right  To  Privacy”  was  authored  by  Warren  and  Brandeis.  (Brandeis  later 
became  a  US  Supreme  Court  Justice.)  They  argued  that  the  law  must  be  extended  beyond  the 
protection  of  property  and  contracts  from  violation  to  include  protection  of  the  “inviolate 
personality”  (Warren  &  Brandeis,  1 890,  p.  205).  They  also  argued  that  invasion  of  privacy  could 
create  mental  pain  greater  than  suffering  inflicted  by  bodily  injury.  Especially  relevant  to  survey 
work,  Warren  and  Brandeis  (pp.  195  &  198)  cited  Judge  Cooley’s  1888  admonition  that  people 
had  the  right  “to  be  let  alone”  and  the  1769  case  of  Millar  v.  Taylor  in  which  it  was  argued  that 
every  man  had  the  right  to  keep  his  own  sentiments  and  to  judge  whether  they  be  made  public. 
They  concluded: 
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“The  common  law  secures  to  each  individual  the  right  of  determining,  ordinarily,  to 
what  extent  his  thoughts,  sentiments,  and  emotions  shall  be  communicated  to  others. 
Under  our  system  of  government,  he  can  never  be  compelled  to  express  them  (except 
when  upon  the  witness  stand);  and  even  if  he  has  chosen  to  give  them  expression,  he 
generally  retains  the  power  to  fix  the  limits  of  the  publicity  which  shall  be  given  them.” 

In  1928,  Justice  Brandeis  reiterated  and  expanded  his  view  in  a  dissenting  opinion  concerning 
the  rights  protected  by  the  Fourth  Amendment.  He  contended  that  the  makers  of  our 
Constitution: 

“sought  to  protect  Americans  in  their  beliefs,  their  thought,  their  emotions  and  their 
sensations.  They  conferred,  as  against  the  Government,  the  right  to  be  let  alone— the 
most  comprehensive  of  rights  and  the  right  most  valued  by  civilized  man.  To  protect 
that  right,  every  unjustifiable  intrusion  by  the  Government  upon  the  privacy  of  the 
individual,  whatever  the  means  employed,  must  be  deemed  a  violation  of  the  Fourth 
Amendment”  (Alderman  &  Kennedy,  1991,  p.l36). 

Alderman  and  Kennedy  (1991,  p  136-137  &  377-378)  report  that  almost  a  half  century  later  the 
Supreme  Court  adopted  this  interpretation  of  the  Fourth  Amendment.  The  Fourth  Amendment 
protects  people,  not  just  places,  from  intrusion  and  an  individual’s  “reasonable  expectation  of 
privacy”  should  be  respected. 

As  Beaney  (1990)  argues,  legal  privacy  rights  are  particularly  important  as  individuals 
and  groups  experience  increasing  and  more  varied  assaults  on  their  inner  life  and  thoughts.  He 
includes  the  widespread  use  of  surveys  (and  by  extension,  interviews)  in  government  and 
industry  as  one  form  of  such  assault.  He  suggests  that  if  Congress  and  government  agencies 
develop  a  heightened  sensitivity  to  potential  threats  to  dignity  and  privacy,  the  need  for  judicial 
intervention  will  decline. 

In  recent  years,  however,  litigation  for  privacy  rights  violations  has  increased 
dramatically.  This  litigation  has  been  initiated  by  employees  against  overly  intrusive  employers 
(see,  for  example,  Alderman  and  Kennedy,  1995,  pp.  273-320,  Privacy  in  the  Workplace).  Much 
privacy  litigation  is  based  on  torts.  Koral  (1988,  p.  3)  defines  a  tort  as  “a  civil  wrong  for  which 
the  law  recognizes  the  victim  is  entitled  to  a  remedy  from  the  person  responsible.”  Lawyers 
have  invoked  common  law  principles  and  federal  and  state  constitutional  provisions  to 
successfully  represent  employee  plaintiffs.  Between  1985  and  mid- 1987  there  were  nearly  100 
privacy  verdicts  against  employers  and  the  average  workplace-privacy  jury  award  during  this 
time  was  $316,000  (Hendricks,  Hayden  and  Novik,  1990;  Nachman  &  Ryan,  1994).  Alderman 
and  Kennedy  (1995,  p.  305)  report  that  “Government  workers  are  protected  by  the  federal  and 
state  constitutions  and  certain  statutes  which  often  enable  them  to  be  more  successful  than  their 
private  sector  counterparts  when  suing  their  employer.”  Thus,  safeguarding  privacy  is  not  only 
an  ethical  necessity;  it  also  makes  good  legal  and  business  sense. 

For  additional  information  on  legal  privacy  rights,  see  Shattuck  (1977),  Hixon  (1987), 
Hendricks,  Hayden  and  Novik  (1990),  Murphy  (1990)  and  Alderman  and  Kennedy  (1991,  1995). 
For  a  discussion  of  privacy,  confidentiality  and  the  necessity  of  ensuring  voluntary  participation 
and  informed  consent  in  ethical  research,  see  Sieber  (1992).  For  a  discussion  in  lay  terms  of 
employee  rights  to  privacy,  see  Schein  (1976),  Koral  (1988)  or  Nachman  and  Ryan,  (1994).  For 
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a  discussion  of  privacy  and  invasiveness  from  the  perspective  of  personnel  selection  instrument 
developers,  see  Mael,  Connerley  and  Morath  (1996). 

Survey  Implications  of  the  Rights  Perspective:  When  the  rights  to  free  speech  and 
privacy  are  combined,  a  few  conditions  for  effective  survey  data  collection  become  clear.  If  you 
want  accurate  information,  especially  about  sensitive  issues,  give  respondents  the  opportunity  to 
not  respond  and  create  conditions  which  encourage  them  to  respond,  despite  possible 
apprehension,  by  protecting  their  privacy.  Thus,  surveys  must  be  voluntary  and  individual 
responses  must  remain  confidential.  Surveys  facilitate  free  expression  by  providing  respondents 
with  a  safe  haven  due  to  their  voluntary  and  confidential  nature.  Attitudes,  beliefs,  values, 
behaviors,  and  background  data  that  respondents  may  have  kept  to  themselves  can  be  expressed 
without  threat  and  without  normative  limitations  if  ethical  principles  are  scrupulously  followed. 
That  which  is  personal  becomes  public  in  grouped  data  to  provide  valuable  information  while 
protecting  privacy  and  preventing  harm. 
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Appendix  D:  American  Psychological  Association  (APA) 

Ethical  Principles  and  Standards  Relevant  to  Surveys 

The  principles  and  standards  described  below  constitute  a  selective  list.  Others 
provided  by  the  APA  also  apply.  Due  to  their  almost  legal  nature,  relevant  parts  have  often  been 
quoted  verbatim.  However,  I  have  also  paraphrased  to  shorten  the  narrative.  Paraphrasing 
occurs  outside  quotation  marks  or  in  parenthetical  expressions.  You  need  not  be  a  member  of 
the  APA  or  a  psychologist  to  follow  these  guiding  principles.  Each  time  you  see  “psychologists” 
substitute  in  your  mind  “survey  developers.” 

General  Principle  D.  Respect  for  People’s  Rights  and  Dignity.  “Psychologists  accord 
appropriate  respect  to  the  fundamental  rights,  dignity  and  worth  of  all  people.  They  respect  the 
rights  of  individuals  to  privacy,  confidentiality,  self  determination,  and  autonomy,  mindful  that 
legal  and  other  obligations  may  lead  to  inconsistency  and  conflict  with  the  exercise  of  these 
rights.  Psychologists  are  aware  of  cultural,  individual,  and  role  differences,  including  those  due 
to  age,  gender,  race,  ethnicity,  national  origin,  sexual  orientation,  disability,  language  and 
socioeconomic  status.  Psychologists  try  to  eliminate  the  effect  on  their  work  of  biases  based  on 
those  factors,  and  they  do  not  knowingly  participate  in  or  condone  unfair  discriminatory 
practices.” 

Note:  The  list  of  cultural,  individual  and  role  differences  included  above  is  used 
elsewhere  in  the  Ethics  Code  and  is  referred  to  hereafter  as  “human  differences.” 

General  Principle  E.  Concern  for  Other’s  Welfare.  “Psychologists  seek  to  contribute 
to  the  welfare  of  those  with  whom  they  interact  professionally.” 

General  Principle  F.  Social  Responsibility.  Psychologists  are  expected  to  be  socially 
responsible.  They  “are  aware  of  their  professional  and  scientific  responsibilities  to  the 
community  and  the  society  in  which  they  work  and  live.”  They  are  “concerned  about  and  work 
to  mitigate  the  causes  of  human  suffering.” 

Ethical  Standard  1 .02.  Relationship  of  Ethics  and  the  Law.  “If  psychologists’  ethical 
responsibilities  conflict  with  law,  psychologists  make  known  their  commitment  to  the  Ethics 
Code  and  take  steps  to  resolve  the  conflict  in  a  responsible  manner.” 

Ethical  Standard  1.08.  Human  Differences.  Where  cultural  and  lifestyle  diversity  may  affect 
work  concerning  individuals  or  groups,  psychologists  get  the  training/consultation  they  need  to 
provide  competent  services  or  they  make  referrals. 

Ethical  Standard  1 .09.  Respecting  Others.  “In  their  work-related  activities,  psychologists 
respect  the  right  of  others  to  hold  values,  attitudes  and  opinions  that  differ  from  their  own.” 

Ethical  Standard  1.10.  Nondiscrimination.  “In  their  work-related  activities,  psychologists  do 
not  in  engage  in  unfair  discrimination”  (based  on  human  differences). 

Ethical  Standard  1.11.  Sexual  Harassment.  “Psychologists  do  not  engage  in  sexual 
harassment.” 
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Ethical  Standard  1.12  Other  Harassment.  “Psychologists  do  not  knowingly  engage  in  behavior 
that  is  harassing  or  demeaning  to  persons  with  whom  they  interact  in  their  work”  (based  on 
human  difference  factors). 

Ethical  Standard  1.14.  Avoiding  Harm.  “Psychologists  take  reasonable  steps  to  avoid  harming 
their  patients  or  clients,  research  participants,  students  or  others  with  whom  they  work.” 

Ethical  Standard  4.02.  Informed  Consent.  “Psychologists  obtain  appropriate  informed 
consent...using  language  that  is  reasonably  understandable  to  participants.”  It  is  obtained  freely 
and  without  undue  influence. 

Ethical  Standard  5.01.  Discussing  the  Limits  of  Confidentiality.  “Psychologists  discuss  (with 
participants  and  clients)..(l)  the  relevant  limitations  on  confidentiality.. ..and  (2)  the  foreseeable 
uses  of  the  information  generated  through  their  services.” 

“The  discussion  of  confidentiality  occurs  at  the  outset  of  the  relationship.” 

“Permission  for  electronic  recording  of  interviews  is  secured  from  clients  and 
(participants).” 

Ethical  Standard  5.02.  Maintaining  Confidentiality.  “Psychologists  have  a  primary  obligation 
and  take  reasonable  precautions  to  respect  the  confidentiality  rights  of  those  with  whom  they 
work  or  consult.” 

Ethical  Standard  5.03.  Minimizing  Intrusions  on  Privacy.  “Psychologists  discuss  confidential 
information  ...only  for  appropriate  scientific  or  professional  purposes  and  only  with  persons 
clearly  concerned  with  such  matters.” 

Ethical  Standard  5.04.  Maintenance  of  Records.  “Psychologists  maintain  appropriate 
confidentiality  in  creating,  storing,  accessing,  transferring  and  disposing  of  records  under  their 
control.” 

Ethical  Standard  5.05.  Disclosures.  “Psychologists  disclose  confidential  information  without 
the  consent  of  the  individual  only  as  mandated  by  law,  or  where  permitted  by  law  for  a  valid 
purpose.. ..Psychologists  also  may  disclose  confidential  information  with  the  appropriate  consent 
of  the  (client),  unless  prohibited  by  law.” 

Ethical  Standard  5.06.  Consultations.  “When  consulting  with  colleagues,  (1)  psychologists  do 
not  share  confidential  information  that  reasonably  could  lead  to  the  identification  of  (anyone) 
with  whom  they  have  a  confidential  relationship  unless  they  have  obtained  the  prior  consent  of 
the  person  or  organization  ....  and  (2)  they  share  information  only  to  the  extent  necessary  to 
achieve  the  purposes  of  the  consultation.” 

Ethical  Standard  5.07.  Confidential  Information  in  Databases.  “If  confidential  information  ... 
is  entered  into  (databases,  etc.)  available  to  persons  whose  access  has  not  been  consented  to  (by 
the  information  provider),  then  psychologists  use  coding  or  other  techniques  to  avoid  the 
inclusion  of  personal  identifiers.” 
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Ethical  Standard  5.08.  Use  of  Confidential  Information  for  Didactic  or  Other  Purposes. 
“Psychologists  do  not  disclose  (in  writings,  lectures,  etc.)  confidential,  personally  identifiable 
information”  without  consent.  They  disguise  such  information  to  prevent  identification  or  harm 
to  the  individuals  involved. 

Ethical  Standard  6.1 1.  Informed  Consent  to  Research.  “Psychologists  use  language  that  is 
reasonably  understandable  to  research  participants  in  obtaining  their  appropriate  informed 
consent.”  ....  “Using  language  that  is  reasonably  understandable  to  participants,  psychologists 
inform  participants  of  the  nature  of  the  research;  they  inform  participants  that  they  are  free  to 
participate  or  to  decline  to  participate  or  to  withdraw  from  the  research;  they  explain  the 
foreseeable  consequences  of  declining  or  withdrawing;  they  inform  participants  of  significant 
factors  that  may  be  expected  to  influence  their  willingness  to  participate ....  and  they  explain 
other  aspects  about  which  the  prospective  participants  inquire.” 

Ethical  Standard  6.12.  Dispensing  With  Informed  Consent.  “Before  determining  that  planned 
research  (such  as  research  involving  only  anonymous  questionnaires,  naturalistic  observations, 
or  certain  kinds  of  archival  research)  does  not  require  the  informed  consent  of  research 
participants,  psychologists  consider  applicable  regulations  and  institutional  review  board 
requirements,  and  they  consult  with  colleagues  as  appropriate.” 

Ethical  Standard  6.13.  Informed  Consent  in  Research  Filming  or  Recording.  “Psychologists 
obtain  informed  consent  from  research  participants  prior  to  filming  them  or  recording  them,” 
unless  the  research  involves  naturalistic  observation  in  public  places  and  the  recording  will  not 
be  used  to  identify  people  or  to  harm  them. 

Ethical  Standard  6.15.  Deception  in  Research.  “Psychologists  never  deceive  research 
participants  about  significant  aspects  that  would  affect  their  willingness  to  participate,  such  as  .... 
unpleasant  emotional  experiences  (etc.).” 

Ethical  Standard  6.16.  Sharing  and  Utilizing  Data.  “Psychologists  inform  research 
participants  of  their  anticipated  sharing  or  future  use  of  personally  identifiable  research  data  and 
of  the  possibility  of  unanticipated  future  uses.” 

Ethical  Standard  6.17.  Minimizing  Invasiveness.  “In  conducting  research,  psychologists 
interfere  with  the  participants  or  milieu  from  which  data  are  collected  (as  little  as  possible).” 

Ethical  Standard  6.18.  Providing  Participants  with  Information  about  the  Study. 
“Psychologists  provide  a  prompt  opportunity  for  participants  to  obtain  appropriate  information 
about  the  nature,  results,  and  conclusions  of  the  research,  and  psychologists  attempt  to  correct 
any  misconceptions  that  participants  may  have.” 

Ethical  Standard  6.19.  Honoring  Commitments.  “Psychologists  take  reasonable  measures  to 
honor  all  commitments  they  have  to  research  participants.” 

Ethical  Standard  6.21.  Reporting  of  Results.  “Psychologists  do  not  fabricate  data  or  falsify 
results.”  They  correct  or  retract  significant  errors. 
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Ethical  Standard  8.03.  Conflicts  Between  Ethics  and  Organizational  Demands.  “If  the 
demands  of  an  organization  ....  conflict  with  this  Ethics  Code,  psychologists  clarify  the  nature  of 
the  conflict,  make  known  their  commitment  to  the  Ethics  Code,  and  to  the  extent  feasible,  seek 
to  resolve  the  conflict  in  a  way  that  permits  the  fullest  adherence  to  the  Ethics  Code.” 


97 


Appendix  £:  Rights  Rationale  For  Not  Asking  For 
Job-Irrelevant  Organizational  Membership 

In  the  text,  I  cite  legal  precedent  for  avoiding  items  about  job- irrelevant  organizational 
membership.  Here,  I  provide  more  elaboration  of  this  view  from  First  Amendment  and  privacy 
rights  perspectives.  The  issue  involves  privacy  of  association  and  belief  which  is  protected  by 
law  because  without  privacy  protection,  first  amendment  rights  could  be  inhibited  (see  Shattuck, 
1977,  Chapter  2,  for  an  extended  discussion  and  numerous  legal  examples).  As  discussed  in 
Appendix  C,  the  Court  has  interpreted  the  Bill  of  Rights  to  include  peripheral  rights.  Thus,  the 
penumbra  of  the  First  Amendment  includes  the  right  to  privacy  of  association  (see  Kauper,  1990, 
pp.  242  &  243). 

Organizational  memberships  can  reflect  the  core  of  an  individual’s  attitudes,  beliefs  and 
values  as  well  as  his  or  her  style  of  First  Amendment  protected  religious,  political  or  social 
expression.  Virtually  every  group  is  going  to  be  unpopular  with  some  people  in  our  pluralistic 
society,  regardless  of  where  they  are  in  the  political  spectrum.  Even  members  of  unpopular 
groups  have  the  right  to  gather  and  to  peacefully  express  their  views.  They  have  the  right  to 
work  within  the  law  for  reform  as  they  see  it.  If  membership  lists  get  into  the  hands  of 
potentially  hostile  parties,  rights  could  be  jeopardized  since  members  could  be  harassed  by  those 
who  object  to  their  expressed  beliefs  or  to  their  lawful  activities.  They  could  be  inhibited  from 
associating  due  to  the  threat  of  public  disclosure.  Thus,  even  in  the  most  democratic  of  societies 
such  as  ours,  democratic  processes  at  the  grass  roots  could  be  stifled. 

In  a  case  described  by  Alderman  and  Kennedy  (1991,  Freedom  of  Assembly:  Hobson  v. 
Wilson,  pp.  69-88),  government  agents  attempted  to  disrupt,  discredit  and  create  discord  among 
members  of  organizations  planning  a  protest  demonstration  in  Washington  DC.  A  jury  found  the 
agents’  actions  unlawful.  The  appellate  court  reduced  the  monetary  award  but  was  clear  in 
stating  that  such  actions  were  intolerable.  It  wrote:  “Whatever  authority  the  Government  may 
have  to  interfere  with  a  group  engaged  in  unlawful  activity  ...  it  is  never  permissible  to  impede  or 
deter  lawful  civil  rights/political  organization,  expression  or  protest  with  no  other  direct  purpose 
and  no  other  immediate  objective  than  to  counter  the  influence  of  the  target  associations”  (p.86). 
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Appendix  F:  Guidance  for  Group  Survey  Administrators 

Be  aware  that  your  survey  may  elicit  a  negative  reaction  from  respondents.  If  people 
express  their  frustration  or  skepticism,  let  them  do  so  freely  and  don’t  take  it  personally.  Any 
hostility  is  not  directed  toward  you.  Surveys  give  respondents  an  opportunity  for  unconstrained 
input  and  they  may  want  to  express  their  displeasure  with  the  survey  itself.  Let  them  vent. 

Respondents  may  also  express  their  displeasure  about  having  to  take  your  survey. 
Surveys  must  always  be  voluntary  and  confidentiality  at  the  level  of  individual  respondents 
must  always  be  maintained.  Group  administration  gives  you  a  good  chance  to  motivate 
participation  since  you  can  talk  to  respondents  and  address  their  concerns.  However,  don’t  force 
anyone  to  take  the  survey  if  they  desire  not  to,  and  don’t  be  nosy  about  how  they  responded  or 
what  comments  they  made.  Emphasize  how  individual  responses  will  not  be  shared  with  their 
supervisors  or  anyone  else  in  their  chain  of  command.  Only  aggregate  (grouped)  data  will  be 
shared  with  management.  If  anyone  in  your  group  of  prospective  participants  does  not  wish  to 
participate,  thank  them  for  coming  and  let  them  leave  without  pressuring  them  or  being 
antagonistic.  Also  note  that  if  someone  agrees  to  take  the  survey  but  doesn’t  like  a  particular 
item,  they  can  skip  the  item.  This  is  not  something  you  generally  communicate  up-front,  but  if 
they  ask,  admit  they  have  the  right  to  skip  items  due  to  the  voluntary  nature  of  the  survey. 

Group  administration  affords  you  with  an  opportunity  to  emphasize  the  importance  and 
purpose  of  the  survey.  Explain  that  it  will  be  used  to  help  rather  than  to  hurt  them.  Be  open  to 
any  questions  they  may  have  and  try  to  allay  their  fears.  If  they  need  to  talk  to  someone  further, 
refer  them  to  the  survey  developers. 

In  a  group  administration  situation  give  people  the  option  of  returning  their  answer  sheet 
and  comments  page  to  survey  administrators  as  they  leave  the  room,  or  to  return  the  survey 
directly  to  the  survey  developer.  Provide  an  envelop  and  even  address  labels  for  those  who  wish 
to  return  the  survey  directly  via  the  mail.  Remind  people  not  to  fold  or  staple  their  answer 
sheets.  As  people  return  their  answer  sheets  and  comments  pages  to  you  do  not  look  at  the 
sheets  or  comments  pages  as  if  you  were  interested  in  how  a  particular  individual 
responded.  You  can  even  have  them  place  their  sheets  and  comments  pages  in  a  pile  rather  than 
have  them  give  these  materials  directly  to  you. 
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Appendix  G:  Statement  of  Work  for  Disk  Duplication  and  Distribution 
From  the  FM  Career  Survey 


DATE  17  May  96 
PAGE  1  of  3 


Statement  of  Work 

TITLE:  Disk  Duplication  and  Distribution  Support  for  USAF  Financial  Management  Career 
Survey 


1 .  Scope:  The  objective  of  this  contract  is  to  support  the  duplication  and  distribution  of 
approximately  4780  automated  USAF  Financial  Management  Career  Surveys  on  3  1/2  inch 
disks,  one  survey  per  disk,  in  bulk,  to  points  of  contact  (POCs)  at  sites  throughout  the  United 
States,  including  Alaska  and  Hawaii.  Addresses  for  people  outside  the  US  will  be  to  stateside 
APOs.  General  instructions,  cover  letters  and  return  mailers  will  be  shipped  with  the  automated 
survey  disks. 

1 . 1  Background.  Since  AL/HRM  has  recognized  expertise  in  survey  development, 
automation  and  analyses,  USAF  SAF/FMBM  requested  our  assistance  with  a  USAF  Financial 
Management  Career  Survey.  This  survey  has  been  developed  and  automated  (i.e., 
computerized)  with  our  assistance,  and  we  can  create  a  data  tape,  analyze  these  data  and 
interpret/brief  results.  However,  AL/HRM  does  not  have  the  manpower  or  capacity  to  easily 
duplicate,  label  and  distribute  several  thousand  disks.  Once  the  automated  survey  is  pretested 
and  refined,  AL/HRM  needs  contractor  assistance  to  duplicate  and  distribute  the  disks  quickly  to 
approximately  4,780  people  in  the  Financial  Management  career  field.  Since  disks  could  be  lost 
in  the  mail,  or  their  may  be  a  need  to  send  out  additional  disks  to  sites  with  a  poor  response  rate, 
up  to  a  grand  total  of  7,000  disks  may  need  to  be  duplicated  and  distributed.  To  obtain 
feedback  quickly  on  important  issues,  this  project  is  on  a  very  fast  schedule:  the  initial  4,780  or 
so  survey  disks  are  expected  to  be  delivered  to  POCs  within  one  week  of  receipt  of  Government 
furnished  material,  except  for  packages  sent  to  APO  addresses  which  will  take  longer  to  reach 
POCs. 


2.  Applicable  or  Reference  Documents.  None. 

3.  Requirements. 

3.1.  Within  three  weeks  of  the  start  of  the  contract,  AL/HRM  or  USAF  SAF/FMBM  will 
overnight  mail  to  the  contractor  a  final,  approved  USAF  Financial  Management  Career  Survey 
on  a  3 1/2  inch  high  density  disk.  The  contractor  shall  duplicate  this  approximately  4780  times 
using  gray  disks.  The  contractor  shall  check  all  disks  for  viruses  and  remove  any  viruses  that 
may  be  present.  The  contractor  shall  label  each  disk.  Labels  shall  be  white  and  of  standard-size. 
Print  shall  be  in  black,  without  a  logo,  with  as  big  a  font  size  as  possible  given  the  size  of  the 
label.  The  text  of  the  label  shall  be  as  follows:  (Note  that  the  title  of  the  survey  is  bold.) 
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USAF  Financial  Management  Career  Survey 
USAF  SAF/FMBM  &  Armstrong  Laboratory 

Go  to  DOS,  insert  disk  into  drive, 
type  letter  for  drive  and  a  colon  (e.g..  A:), 
hit  enter  and  type  “survey” 

3.2.  USAF  SAF/FMBM  will  provide  the  contractor  with  a  list  of  sites,  with  the  names, 
addresses  and  phone  numbers  of  survey  monitors  or  points  of  contact  (POCs)  and  the  number  of 
disks  to  be  sent  to  each  site.  POCs  at  up  to  70  sites  will  receive  survey  disks.  At  least  half  of 
these  sites  will  be  receiving  a  large  (but  as  yet  unknown)  number  of  disks.  As  more  information 
is  know  it  will  be  provided  to  PK  or  potential  contractors  to  aid  in  cost  estimation.  Addresses 
will  be  quite  specific  and  include  a  street  address  and  room  number  wherever  possible.  This 
information  will  be  provided  to  the  contractor  on  a  3  1/2  inch  disk  as  a  Microsoft  Word  or 
Microsoft  Excel  document.  In  addition  to  POC  addresses  and  other  data,  this  information  disk 
will  include  general  instructions  to  POCs  and  a  cover  letter  for  respondents.  The  first  of  these 
Government-prepared  documents  will  explain  the  role  of  the  POC,  the  purpose  of  the  survey,  and 
provide  POCs  with  specific  information  about  how  to  distribute  the  disks  and  how,  where  and  by 
when  they  should  be  returned.  The  second  document  will  be  quite  similar  to  the  cover  letter 
already  included  at  the  beginning  of  the  automated  survey.  The  contractor  shall  print  these 
materials  from  the  disk  and  copy  them  as  needed.  The  contractor  shall  package  the  disks  for 
distribution  to  each  site  (the  number  of  disks  varying  by  site),  along  with  one  paper  copy  of  the 
general  directions  to  POCs  and  one  paper  copy  per  disk  of  the  cover  letter  for  respondents.  This 
shall  be  done  in  a  manner  to  ensure  disks  and  other  materials  reach  their  destination  undamaged. 
If  damage  is  reported,  the  contractor  shall  send  replacement  materials.  The  contractor  shall 
duplicate  additional  disks  and  replace  and  reship  any  lost  packages.  If  directed  by  the 
Government  the  Contractor  shall  also  duplicate  and  ship  additional  disks  to  selected  sites  (gain 
up  to  the  grand  total  of  7,000  disks)  if  response  rate  at  any  site  is  found  to  be  poor  (less  than  50% 
return).  If  this  is  necessary,  address  label  each  package  and  make  shipping  arrangements  to 
ensure  that  the  packages  arrive  at  their  destination  within  72  hours  or  3  days.  (Packages  shipped 
to  APO  addresses  will  take  a  few  weeks  to  reach  their  final  destination.  Therefore  packages  for 
APO  destinations  shall  just  reach  their  APO  addresses  within  3  days.)  Packages  shall  be  shipped 
in  a  manner  that  allows  the  shipping  route,  the  time  of  arrival,  and  the  receiving  person  to  be 
monitored  in  case  a  shipment  gets  lost.  The  contractor  shall  duplicate  additional  disks  and 
replace  and  reship  any  lost  packages  if  they  are  unable  to  be  found  and  delivered  within  24  hours 
of  the  “3  days  to  destination”  requirement.  If  necessary,  the  Contractor  shall  also  duplicate  and 
ship  additional  disks  to  selected  sites  (again  up  to  the  grand  total  of  7,000  disks)  if  the  response 
rate  at  any  site  is  found  to  be  poor  (less  than  50%).  If  this  is  necessary,  the  costs  of  duplicating 
the  disks  and  shipping  them  once  again  to  POCs  will  be  at  Government  expense.  The  contractor 
shall  produce,  virus  check,  package  and  ship  all  diskettes  such  that  they  are  received  by  the 
POCs  no  later  than  one  week  after  receiving  Government  furnished  materials  identified  in 
paragraph  5  (except  packages  sent  to  APO  addresses  which  will  take  longer  to  reach  POCs). 
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3.3.  In  addition  to  shipping  the  disks,  POC  instructions,  and  a  cover  letter  in  the 
shipments  described  in  paragraph  3.2,  the  contractor  shall  include  in  these  packages  a  labeled  6” 
by  6”  return  mailer  for  every  disk  shipped.  Each  return  mailer  shall  be  labeled  with  the 
following  address: 


Armstrong  Laboratory 
AL/HRMJ  (Attn:  Dr.  Tom  Watson) 

7909  Lindbergh  Drive 
Brooks  AFB,  TX  78235-5352,  USA 

4.  Base  Support.  None. 

5.  Government  Furnished  Property.  AL/HRM  or  USAF  SAF/FMBM  will  supply  the 
contractor  with  the  following  four  items  on  two  disks:  1)  a  final  approved  automated  survey  on  a 
separate  disk,  2)  detailed  site,  POC  address  and  phone  information,  3)  general  instructions  to 
POCs  and  4)  a  cover  letter  for  respondents.  The  last  three  items  will  be  on  a  separate 
information  disk. 

6.  Data  Requirements.  None. 

7.  Points  of  Contact 

7.1  AL: 

AL/HRMJ  (Attn:  Dr.  Tom  Watson,  Room  3 1 9,  phone  (2 1 0)  536-3640  or  DSN  240-3640) 

7909  Lindbergh  Drive 
Brooks  AFB,  TX  78235-3640 

7.2  FM: 

USAF  SAF/FMBM  (Attn:  Mr.  Robert  Zook  ,  Room  4  D  120,  phone  (703)  697-1 120  or  DSN 
227-1120) 

1130  Air  Force  Pentagon 
Washington  DC  20330-1 130 
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Appendix  H:  Guidance  for  Automated  Survey  Points  of  Contact 
From  the  FM  Career  Survey 

DEPARTMENT  OF  THE  AIR  FORCE 
HEADQUARTERS  UNITED  STATES  AIR  FORCE 
WASHINGTON  DC 

Guidance  for  FM  Career  Survey  Points  of  Contact 

June  1996 

You  have  been  selected  as  a  point  of  contact  to  help  distribute  an  automated  (3.5” 
diskette)  AF  Financial  Management  Career  Survey,  with  a  memorandum  to  respondents  and  a 
return  mailer,  to  people  on  your  base  or  in  your  general  vicinity.  A  list  of  selected  respondents 
and  their  organizational  symbols  is  also  enclosed.  Disks  and  cover  memos  have  been  placed  in 
each  return  mailer  to  make  distribution  easier.  Each  disk  contains  an  almost  identical  copy  of 
the  cover  memo  and  one  survey.  This  survey  is  designed  to  assess  the  attitudes  and  knowledge 
of  military  members  and  civilian  employees  worldwide  on  a  variety  of  career  management 
issues.  A  sample  of  about  4,700  FM  people  will  be  sent  this  survey,  some  of  whom  are  at  your 
site.  Please  distribute  these  disk  packets  to  the  people  whose  names  are  on  the  list  within  the 
next  few  days  and  urge  them  to  respond  rapidly—within  five  days  or  less.  You  should  be  able  to 
reach  most  people  through  the  base  distribution  system.  We  have  worked  hard  to  avoid  sending 
you  disks  that  would  require  re-mailing.  If  the  intended  recipient  is  not  available  (TDY,  PCS, 
leave),  please  pass  the  survey  to  someone  of  equivalent  or  similar  grade  and  functional 
discipline.  Respondents  should  use  the  pre-addressed  return  mailer  to  send  completed  surveys 
directly  back  to  the  Air  Force’s  Armstrong  Laboratory  for  analyses. 

This  survey  is  voluntary.  We’d  like  you  to  encourage  people  to  complete  it,  but  no  one 
should  be  forced  to  do  so  or  be  told  the  survey  is  mandatory.  If  an  intended  recipient  does  not 
wish  to  participate,  pass  the  disk  and  mailer  to  someone  of  similar  grade  and  functional 
discipline.  The  survey  is  also  confidential.  Each  disk  contains  no  personal  identifier  and  none 
is  requested  except  for  a  self-generated  password  used  to  discourage  unauthorized  viewing. 
Requested  background  information  is  intentionally  broad  to  prevent  identification.  Individual 
responses  will  not  be  reported  to  anyone  in  a  respondent’s  chain  of  command  (except  verbatim 
comments  respondents  may  provide).  Only  grouped  data  will  be  used  for  analyses  in  which 
individual  responses  are  combined  with  those  of  others  for  total  sample  or  subgroup  analyses. 
Subgroups  will  be  formed  on  the  basis  of  responses  to  the  background  items  (such  as  grade  or 
military/civilian  status)  in  the  survey. 

Please  read  the  cover  letter  carefully.  We  would  like  you  to  be  familiar  with  the 
importance  of  the  survey  to  encourage  people  to  take  it.  The  survey  offers  people  a  chance  to 
communicate  directly  with  the  senior  leadership  on  important  career  management  issues.  Senior 
Leaders  need  responses  from  across  the  FM  career  field  to  make  timely  decisions  to  improve 
career  management  practices.  We  would  like  responses  to  be  sent  back  within  five  days  so  we 
can  begin  data  analyses,  find  out  what  our  people  think  are  important  issues,  and  develop 
strategies  for  improving  career  field  management. 

Getting  into  the  survey  is  easy:  go  into  DOS,  insert  the  disk  into  the  drive,  type  the 
drive  designation  (A,  for  example)  followed  by  a  colon  (:),  hit  enter  and  type  “survey.” 
Respondents  can  also  go  through  their  file  manager  to  the  desired  drive  and  click  on 
“survey.bat.”  Other  directions  are  provided  in  the  survey. 
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Feel  free  to  follow  up  with  survey  recipients  to  see  if  they  have  completed  and  returned 
the  surveys.  If  they  haven’t,  please  encourage  them  to  reply  without  requiring  it  since  the  survey 
is  voluntary.  If  respondents  return  disks  to  you,  you  may  return  them  in  bulk  to  the  address  that 
follows.  We  would  prefer  individual  respondents  to  return  the  surveys  directly  to  the  Armstrong 
Laboratory  in  the  mailers  provided.  Return  any  completed  survey  disks  to: 

Armstrong  Laboratory 

AL/HRMJ  (Attn.:  Dr.  Tom  Watson,  Room  319) 

7909  Lindbergh  Drive 
Brooks  AFB,  TX  78235-5352,  USA 

Thanks  for  your  support!  Your  involvement  makes  the  process  easier  and  gives  it 
important  visibility.  If  you  or  others  have  questions  or  concerns,  please  contact  Major  Ron 
Herpst,  SAF/FMBM,  at  DSN  227-5627,  or  Dr.  Tom  Watson,  Armstrong  Laboratory,  at  DSN 
240-3640.  If  you  are  having  problems  with  survey  automation,  please  contact  SSgt.  David 
Selander,  Armstrong  Laboratory,  DSN  240-1 166.  Thanks  again  for  your  help. 

ROBERT  W.  ZOOK 

Director  of  Budget  Management  and  Execution 

Deputy  Assistant  Secretary  (Budget) 


Enclosures 

1 .  Disk  packets,  each  containing  1  survey  disk,  1  memorandum  to  respondents  and  1  return 
mailer 

2.  List  of  names  and  organizational  symbols  of  intended  respondents 
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Appendix  I:  Examples  of  Likert-Style  Response  Scales 
(Also  see  the  agreement  scale  in  the  text  on  page  39.) 

Example  1:  6-Point  Satisfaction  Scale  With  No  Neutral  Midpoint 

Use  the  scale  below  to  indicate  your  satisfaction  or  dissatisfaction  with  the  following  issues.  If  a 
statement  is  not  applicable  to  you,  mark  response  “G.” 

A  B  C  D  E  F  G 


Very  Dissatisfied  Somewhat  Somewhat  Satisfied  Very  Not  Applicable 

Dissatisfied  Dissatisfied  Satisfied  Satisfied 

Note:  You  could  also  use  a  Much  More  Satisfied  to  Much  Less  Satisfied  scale  if  you  wanted  to 
contrast  satisfaction  before  some  intervention  with  satisfaction  after  it.  See  Example  8  in  this 
appendix.  Note:  since  this  had  to  do  with  personal  satisfaction  concerning  different  facets  of  the 
workplace,  “Don’t  know”  was  not  used.  People  could  be  expected  to  know  their  own  personal 
satisfaction  with  aspects  of  the  work  environment.  Not  Applicable  may  also  not  make  sense,  but 
was  left  to  provide  an  escape  option. 

Example  2:  6-Point  Extent  Scale  With  No  Neutral  Midpoint 

Use  the  scale  below  to  indicate  to  what  extent  the  following  statements  about  your  medical 
treatment  facility  are  true.  If  you  don’t  know,  or  a  statement  is  not  applicable,  mark  response 
“G.” 

A  B  C  D  E  F  G 


Not  at  To  a  Very  To  a  Little  To  Some  To  a  Great  To  a  Very  Don’t  Know  or 
All  Little  Extent  Extent  Extent  Extent  Great  Extent  Not  Applicable 

Example  3:  6-Point  Importance  Scale  With  No  Neutral  Midpoint 

Use  the  scale  below  to  indicate  how  important  each  of  the  following  factors  would  be  (for  some 
purpose).  If  you  don’t  know,  or  a  statement  is  not  applicable  to  you,  mark  response  “G.” 

A  B  C  D  E  F  G 


Very  Unimportant  Somewhat  Somewhat  Important  Very  Don’t  Know  or 
Unimportant  Unimportant  Important  Important  Not  Applicable 
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Example  4:  6-Point  Willingness  Scale  With  No  Neutral  Midpoint 

Use  the  scale  below  to  indicate  your  willingness  (to  do  something).  If  you  don’t  know,  or  a 
statement  is  not  applicable  to  you,  mark  response  “G.” 

A  B  C  D  E  F  G 


Very  Unwilling  Somewhat  Somewhat  Willing  Very  Don’t  Know  or 
Unwilling  Unwilling  Willing  Willing  Not  Applicable 

Example  5:  6-Point  Yes/No  Scale  With  No  Neutral  Midpoint 

Use  the  scale  below  to  indicate  whether  you  would  or  would  not  enjoy  participating  in  the 
following  activities.  If  you  don’t  know,  or  a  statement  is  not  applicable  to  you,  mark  response 


A  B  C  D  E  F  G 


Definitely  No  Probably  Probably  Yes  Definitely  Don’t  Know  or 
No  No  Yes  Yes  Not  Applicable 

Example  6:  5-Point  Agreement  Scale  With  a  Neutral  Midpoint 

Scales  need  not  have  the  six  points  I  typically  use.  And  if  you  wish,  you  may  use  a  neutral 
midpoint.  Below  are  three  examples.  Note  that  for  the  satisfaction  scales  (as  in  Example  1),  I 
did  not  include  a  “Don’t  Know”  as  part  of  the  escape  option  because  people  usually  know  if  they 
are  satisfied  or  not  with  different  facets  of  the  workplace. 

Use  the  scale  below  to  indicate  your  agreement  or  disagreement  with  each  of  the  following 
statements.  If  you  don’t  know,  or  a  statement  is  not  applicable  to  you,  mark  response  “F.” 


A 

BCD 

E 

F 

Strongly 

Disagree  Neither  Agree  Agree 

Strongly 

Don’t  Know  or 

Disagree 

Nor  Disagree 

Agree 

Not  Applicable 
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Example  7:  4-Point  Agreement  Scale  With  No  Neutral  Midpoint 

You  may  need  an  abbreviated  scale,  for  instance  on  a  telephone  survey  where  the 
respondent  hears  but  does  not  see  the  verbal  anchors. 

A  B  C  D  E 


Strongly  Somewhat  Somewhat  Strongly  Don’t  Know  or 

Disagree  Disagree  Agree  Agree  Not  Applicable 

Example  8:  5-Point  Satisfaction  Scale  With  a  Neutral  Midpoint 

Use  the  scale  below  to  indicate  your  satisfaction  or  dissatisfaction  with  the  following  issues.  If  a 
statement  is  not  applicable  to  you,  mark  response  “F.” 


Veiy  Somewhat  Neither  Satisfied  Somewhat  Very  Not  Applicable 

Dissatisfied  Dissatisfied  Nor  Satisfied  Satisfied 

Dissatisfied 

Example  9:  5-Point  Satisfaction  Scale  With  a  Neutral  Midpoint 
For  Post  Intervention  Comparison 

Use  the  scale  below  to  indicate  your  satisfaction  now  compared  to  your  satisfaction  prior  to  the 
intervention.  If  a  statement  is  not  applicable  to  you,  mark  response  “F.” 

A  B  C  D  E  F 


Much  Less  Somewhat  Less  Neither  More  Somewhat  More  Much  More  Not  Applicable 

Satisfied  Satisfied  Nor  Less  Satisfied  Satisfied 

Satisfied 

Example  10:  5-Point  Excellence  Scale 

I  was  recently  asked  to  suggest  an  “Excellent-Poor”  scale  for  ranking  customer  service  using  a 
telephone  survey.  Due  to  the  need  to  keep  the  scale  simple  (since  verbal  anchors  had  to  be 
remembered  rather  than  observed  by  the  respondent),  I  suggested  use  of  the  following  simple  5- 
point  scale: 

How  would  you  rate  the  quality  of  service  provided  to  you  at  the  (whatever)? 

A  B  C  D  E  F 


Excellent  Good  Fair  Poor  Unsatisfactory  Not  Applicable 
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Appendix  J:  Guidance  on  Interviewing 

(Extracted  from  Watson,  1997,  Guidelines  for  Conducting  Interviews) 

The  Advantages  of  Interviews 

If  you  need  richer  information  than  is  possible  using  a  survey,  you  can  get  it  from  a 
small  number  of  people,  and  you  are  not  concerned  about  being  able  to  perform  sophisticated 
quantitative  data  analysis,  interviewing  may  be  your  preferred  method.  Relative  to  surveys, 
interviews  have  a  number  of  advantages,  as  discussed  below. 

You  Usually  are  Face-to-Face:  Relative  to  paper-and-pencil  mail  surveys,  or  telephone 
interviews,  person-to-person  interviews  are  face-to-face.  They  are,  therefore,  more  intimate. 
They  provide  interviewers  with  the  opportunity  to  establish  a  brief  social  relationship,  to  observe 
social  dynamics  and  use  them  to  good  advantage,  and  to  observe  verbal  and  non-verbal  cues. 
Interviewers  can  bond  with  those  being  interviewed.  If  interviewers  are  kind,  sincere,  and 
interested  in  what  interviewees  have  to  say,  they  may  be  liked  and  trusted,  creating  a  situation 
where  people  feel  free  to  talk.  Being  face-to-face  facilitates  the  process  of  getting  people  to  open 
up.  Being  present  with  those  who  are  interviewed  also  gives  interviewers  a  chance  to  observe 
how  interviewees  interact— with  interviewers,  colleagues,  friends,  family,  or  strangers.  Much  can 
be  learned  from  non-verbal  cues,  such  as  posture,  facial  expressions,  hand  gestures  and  level  of 
nervousness  or  threat.  Such  input  would  not  be  available  from  a  survey  and  it  would  be  more 
camouflaged  in  a  telephone  interview.  The  face-to-face  interview  helps  interviewers  detect 
reactions  ranging  from  acceptance  and  liking  to  mistrust  and  cynicism.  Interviewers  can  also 
gain  a  sense  of  openness  and  honesty  or  defensiveness  and  deception.  They  can  also  sense  the 
strength  and  character  of  participant  reaction  to  the  content  of  the  interview— emotionally  and 
intellectually. 

Interviewees  are  Unconstrained  in  How  to  Respond:  In  interviews,  unlike  in  surveys, 
respondents  get  to  answer  questions  in  their  own  words.  They  can  also  define  the  question  and 
shape  the  answer  quite  broadly.  Answers  may  be  more  difficult  to  quantify,  but  the  information 
obtained  is  richer  and  more  complete  than  the  knowledge  derived  from  answers  to  survey  items. 
In  addition,  participants  can  provide  verbal  narrative  responses  to  interview  questions  more 
easily  than  if  they  had  to  write  lengthy  narratives  on  a  survey  comments  page. 

Because  responding  is  unconstrained,  interviews  are  sometimes  similar  to  therapy 
sessions.  Participants  can  ventilate  their  feelings  and  frustrations  in  a  supportive,  protective 
environment.  This  can  help  them,  while  providing  the  interviewer  with  rich  information. 

You  Can  Clarify  and  Probe:  In  an  interview,  unlike  on  a  survey,  the  interviewer  can 
follow  up  immediately  on  what  a  person  says  to  clarify  and  probe  for  greater  understanding.  It  is 
this  process  of  exploration  that  allows  the  interviewer  to  tailor  questions  to  responses  given  to 
get  a  fuller  understanding  of  what  participants  are  trying  to  say. 

You  Are  In  A  Better  Position  to  Motivate  Participation  and  Establish  Rapport: 

Your  role  as  an  interviewer  is  not  only  to  collect  information,  but  to  motivate  participation. 
Because  interviewers  are  usually  face-to-face,  they  are  in  a  better  position  than  a  survey 
developer  to  explain  why  participation  is  so  important  and  to  persuade  reluctant  respondents  to 
take  part. 
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Interviewer  Skills  and  Characteristics 


Skills  Required  of  Interviewers 

Interviewing  requires  great  skill— both  interpersonal  and  cognitive.  Prior  to  the 
interview,  the  interviewer  must  be  able  to  develop  a  good  sampling  methodology  and  generate 
good  interview  questions.  These  tasks  require  good  research  skills.  In  addition,  the  interviewer 
must  make  arrangements  with  organizations,  groups  and  individuals  to  conduct  interviews.  This 
requires  research  skills,  political  savvy  and  the  ability  to  relate  well  to  strangers. 

Skills  required  during  the  interview  are  particularly  numerous.  In  the  interview,  the 
interviewer  must  be  able  to  do  the  following: 

•  establish  rapid  rapport  and  motivate  participation, 

•  establish  trust, 

•  overcome  unwillingness  to  speak  freely  about  sensitive  topics 

•  honor  the  right  to  not  respond  and  avoid  undue  pressure, 

•  value  respondents  regardless  of  the  beliefs  or  behaviors  they  divulge, 

•  listen  actively  and  intently, 

•  speak  effectively  and  persuasively,  but  not  too  much, 

•  interpret  and  respond  to  a  variety  of  different  verbal  and  nonverbal  cues, 

•  be  attentive  to  both  the  interview  process  and  content, 

•  encourage  everyone  to  speak, 

•  maintain  non-coercive  control  over  the  interview  process, 

•  accurately  record  what  is  said, 

•  transition  effectively  from  one  question  or  topic  to  another, 

•  insightfully  interpret,  explore,  link  and  paraphrase  incoming  information, 

•  maintain  neutrality,  keeping  personal  biases  and  beliefs  outside  the  interview, 

•  avoid  leading  questions  or  distortion  of  responses  through  reinforcement, 

•  take  possible  abuse  without  becoming  flustered, 

•  be  kind,  considerate,  and  professional  at  all  times,  and 

•  know  when  and  how  to  terminate  the  interview. 

After  the  interview,  interviewers  must  be  able  to  complete  documentation,  analyze  and 
interpret  the  results,  and  report  results  in  aggregate  form  to  audiences  ranging  from  those 
interviewed  to  academicians.  Interviewers  must  also  be  able  to  maintain  confidentiality  by  not 
disclosing  the  sources  of  comments  despite  possible  pressure.  Selecting  interviewers  who 
already  possess,  or  who  can  develop  these  skills,  and  training  them  well  is  critical. 

Desirable  Personal  Characteristics  of  Interviewers 

Some  people  would  have  difficulty  being  good  interviewers  even  with  training.  Others 
have  characteristics  that  suit  them  well  to  interviewing  that  may  be  enhanced  through  training. 
Below  are  personal  characteristics  I  consider  essential  to  a  good  interviewer. 

Ability  to  Relate  Well  Based  on  Genuine  Caring:  One  essential  characteristic  is  the 
ability  to  relate  well  to  a  variety  of  people  with  whom  you  can  quickly  establish  rapport.  You 
need  to  have  the  interpersonal  skill  to  draw  people  out  so  they  will  communicate  with  you  and 
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with  each  other.  You  relate  based  on  a  heartfelt  concern  for  the  people  being  interviewed  and  a 
desire  to  understand  what  they  have  to  say.  Thus,  you  need  to  be  genuinely  warm,  friendly  and 
interested  in  the  well  being  and  views  of  those  you  interview. 

Genuine  caring  connotes  concern  for  the  welfare  of  those  you  interview  regardless  of 
their  expressed  views  or  reported  behaviors.  You  may  personally  disagree  with  them  and  not 
condone  their  behaviors.  However,  this  does  not  diminish  your  concern  for  their  well  being. 

Kindness  and  Courtesy:  Related  to  the  first  point,  you  have  to  be  kind  and  courteous 
and  be  able  to  treat  everyone  interviewed  with  equality,  dignity,  and  respect,  even  if  you  find 
their  views  or  behaviors  offensive. 

Empathy:  To  interview  well  you  must  tune  into  another  person’s  thoughts  and  feelings. 
This  is  the  ability  to  experience  and  understand  at  an  intellectual  and  emotional  level  what 
another  person  is  telling  you,  as  if  you  were  that  person.  However,  while  doing  so  you  remain  in 
touch  with  your  own  sense  of  self.  Being  empathic  is  like  the  ability  of  an  actor  to  become  the 
person  he  or  she  portrays.  Empathy  probably  facilitates  the  willingness  of  others  to  open  up  to 
you. 


Humility:  Humility  and  a  willingness  to  not  steal  the  show  are  essential.  You  may  be 
the  interview  expert,  but  those  you  are  interviewing  are  the  content  experts  and  without  their 
feedback  you  would  know  little  or  nothing  about  the  topic  you  are  exploring. 

Integrity:  If  you  have  integrity,  you  can  be  trusted.  Thus,  if  you  promise  to  keep 
individual  responses  confidential  and  not  use  what  is  said  to  hurt  anyone,  people  will  believe  you 
and  open  up  to  you.  To  a  limited  extent  you  may  be  able  to  convey  integrity  during  the 
interview.  Over  time,  if  you  develop  a  reputation  for  integrity,  your  credibility  as  an  interviewer 
will  increase. 

Enthusiasm,  Calmness  and  Flexibility:  You  must  be  enthusiastic  enough  to 
communicate  the  importance  of  the  interview  and  your  interest  in  it.  Yet  you  need  to  be  calm 
and  able  to  take  abuse  while  maintaining  subtle  control  without  getting  angry.  Also,  you  should 
be  flexible  to  deal  comfortably  with  the  unexpected  and  accept  things  not  going  as  you  planned. 
In  addition,  you  need  to  be  able  to  listen  to  and  understand  content  you  may  disagree  with.  You 
must  also  be  able  shift  course  based  on  interviewee  reaction  or  the  content  of  the  discussion. 

Tolerance  and  Open-Mindedness:  It  is  essential  to  be  tolerant  of  others  and  open- 
minded  rather  than  judgmental.  People  will  not  communicate  freely  and  you  will  have  difficulty 
understanding  their  perspective  if  you  are  intolerant  of  their  views  or  behaviors.  You  need  to 
value  cultural  diversity.  Above  all,  you  need  to  be  able  to  refrain  from  letting  your  own  views  or 
values  interfere  with  the  interview  process  and  your  understanding  of  what  is  said.  During  an 
interview  you  must  not  be  dogmatic.  You  must  not  try  to  force  your  views  on  others  or  to  be 
critical  of  what  they  say.  However,  your  own  views  and  values  can  help  you.  Do  you  believe 
people  should  express  themselves  freely  and  participate  in  decision  making?  Do  you  value 
understanding  the  views  or  behaviors  of  others,  even  if  divergent  from  your  own?  Do  you  value 
gathering  information  to  help  others?  If  so,  your  skills  as  an  interviewer  should  be  enhanced. 

Assertiveness,  Articulation  and  Ability  to  Listen:  You  need  to  have  the  contradictory 
qualities  of  assertive  articulation  and  the  ability  to  be  quiet  and  listen.  You  must  have  a  good 
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sense  concerning  when  to  switch  from  one  mode  to  the  other,  and  you  must  be  comfortable 
listening  most  of  the  time.  You  also  need  to  be  able  to  reflect  back  what  has  been  said,  to 
express  your  empathy  and  to  provide  a  veracity  check  on  your  understanding. 

Tolerance  of  Ambiguity,  Intuitiveness  and  Abstract  Conceptualization: 

Interviewing  is  an  inductive  process  where  pattern  and  meaning  emerge  from  data  rather  than 
being  imposed  on  them  beforehand.  As  an  interviewer,  you  need  to  be  tolerant  of  ambiguity 
since  you  are  constantly  fitting  the  pieces  of  an  incomplete,  sometimes  contradictory,  puzzle 
together.  You  need  to  be  intuitive  and  insightful,  synthesizing  and  integrating  more  than 
analyzing,  while  fitting  diverse,  seemingly  unrelated,  bits  of  information  together.  You  must  be 
able  to  think  abstractly,  eventually  putting  the  diverse,  incomplete,  pieces  of  information 
together  to  see  the  “big  picture.” 

Speaking  freely,  maintaining  privacy  and  voluntary  participation:  Interviewers  encourage 
unconstrained  speech  to  gain  a  thorough  understanding  of  attitudes,  beliefs,  values  and 
behaviors.  Since  speech  becomes  so  unconstrained,  and  since  interviews  are  intrusive,  the 
privacy  of  respondents  should  be  protected  and  confidentiality  should  be  guaranteed  to  prevent 
harm.  If  information  about  specific  individuals  is  disclosed,  it  should  be  with  their  consent. 

Voluntary  Participation:  Potential  respondents  should  be  given  the  right  not  to 
participate.  Interviewers  motivate  voluntary  participation  by  creating  a  “safe  haven”  so  that 
most  potential  respondents  will  feel  comfortable  enough  to  participate.  Potential  participants 
will  hopefully  want  to  respond  (not  just  feel  comfortable  doing  so)  if  the  interviewer  conveys  the 
importance  of  the  interview  and  explains  the  positive  outcomes  that  are  likely  to  accrue. 

Maintain  Confidentiality  and  Non-Attribution:  There  is  a  distinction  between 
anonymity  and  confidentiality.  Anonymity  is  best,  but  cannot  always  be  promised.  There  is  a 
subtle  “can  not”  versus  “will  not”  distinction.  If  a  survey  is  anonymous,  no  information  is 
collected  which  could  link  data  to  individual  respondents.  Thus,  a  breach  of  privacy  need  not  be 
feared.  However,  it  is  the  author’s  experience  that  many  people  do  not  trust  that  anonymous 
surveys  are,  in  fact,  anonymous.  Many  people  believe  survey  developers  have  some  trick  way  to 
identify  them.  Perhaps  this  distrust  is  understandable  because  survey  researchers  often  can  link 
data  back  to  individual  respondents  or  small  groups.  They  can  do  so  directly  via  an  assigned 
code  or  indirectly  via  demographic  (background)  information.  Respondents  are  often  savvy 
enough  to  realize  you  can  identify  them  through  background  data,  and  even  grouped  data  can  be 
used  to  identify  them.  For  instance,  during  data  analysis,  you  may  discover  a  subgroup’s  “cell 
size”  is  only  one  or  two  people.  Do  not  claim  that  respondents  can  only  be  identified  through 
their  name,  social  security  number  or  survey  code  number.  If  confidentiality  is  promised, 
researchers  basically  imply  or  admit  they  can  tie  individuals  to  responses,  but  promise  they  will 
not.  Since  some  people  don’t  even  trust  anonymous  surveys,  it  has  been  the  author’s  approach  to 
ignore  the  anonymity  versus  confidentiality  distinction  and  to  not  even  argue  whether 
respondents  can  be  identified.  He  just  stresses  confidentiality  and  promises  that  individual  data, 
except  for  verbatim  comments,  will  not  be  released.  Over  the  years,  he  and  his  colleagues  have 
gained  the  reputation  they  can  be  trusted. 

In  most  interviews,  unlike  in  survey  research,  anonymity  would  be  difficult  to  achieve. 
Anonymity  means  no  data  exists  to  link  responses  to  individuals.  In  face-to-face  interviews,  the 
interviewer  sits  with  respondents  whose  identities  are  usually  obvious.  In  a  telephone  interview, 
the  phone  number  could  be  used  to  identify,  at  least,  the  household  or  office  called.  Thus, 
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maintaining  confidentiality  (not  linking  data  to  individuals  even  though  you  could)  is  especially 
critical  for  interviewers.  The  phrase  “not  for  attribution”  is  often  used,  meaning  that  comments 
will  not  be  attributed  to  the  individuals  making  them. 

Assistance  From  Points  of  Contact:  You  may  arrange  for  assistance  from  interview  points  of 
contact  (POCs)  at  different  interview  locations.  They  can: 

•  help  gain  the  cooperation  or  “buy-in”  of  management  at  their  site, 

•  help  you  identify  those  to  be  interviewed, 

•  schedule  the  interviews,  and 

•  arrange  for  a  suitable  location  for  the  interviews  to  take  place. 

Basically,  you  provide  POCs  with  guidance  and  they  identify  your  sample  and  set  up  interviews 
for  you.  The  role  of  POCs  is  important  and  it  is  helpful  to  establish  a  good  relationship  with 
them.  POCs  may  arrange  for  your  lodging  and  serve  as  your  host  and  problem  solver.  They  may 
locate  no-shows  and  reschedule  participants  while  you  are  interviewing.  They  are  likely  to 
become  interview  participants  themselves.  On  the  down  side,  you  may  not  know  for  sure  if  your 
sampling  guidance  was  followed.  Also,  their  dual  role  as  POC  and  interviewee  may  place  them 
in  an  awkward  position.  They  may  be  considered  closely  tied  to  management  and  a  potential 
threat  to  confidentially. 

Group  Interviews:  Group  interviews  allow  multiple  people  to  express  their  thoughts  about  the 
issues  discussed.  They  save  time  since  several  people  are  interviewed  at  once.  They  provide 
input  greater  than  the  sum  of  individual  contributions  had  each  person  been  interviewed  alone. 
Synergy  is  created  when  people  are  influenced  by  and  react  to  the  comments  of  others. 

Group  interviews  can  create  problems,  especially  in  hierarchical  structures.  If  great 
disparities  in  grade  or  rank  exist,  or  if  the  head  of  a  group  is  present,  this  could  result  in 
domination  by  those  who  are  higher  graded  or  in  a  position  of  authority.  The  presence  of  such 
people  could  also  intimidate  those  lower  in  the  hierarchy.  For  this  reason,  it  is  sometimes  wise 
to  exclude  mid-  or  executive-level  supervisors  from  group  interviews,  and  to  request  everyone  be 
of  about  the  same  grade  or  rank.  Of  course,  some  people  may  try  to  dominate  or  intimidate 
regardless  of  their  rank  or  position.  If  you  encounter  domination  or  intimidation  fi'om  any 
source,  engage  others  in  the  discussion  and  do  what  you  can  to  lessen  these  effects. 

Structure:  Some  group  interviews  are  quite  unstructured,  allowing  participants  to  interact 
rather  freely  with  themselves  and  the  interviewer.  An  example  of  such  a  group  is  called  a  focus 
group  (see  Krueger,  1988;  Morgan,  1988;  Stewart  &  Shamdasani,  1990).  Focus  groups  are  used 
extensively  in  marketing  research  and  for  other  purposes.  In  such  a  group,  about  7  to  10  people 
get  together  to  share  their  views  on  a  product,  service  or  process.  These  groups  are  intentionally 
permissive  to  elicit  a  wide  array  of  opinions.  There  is  no  need  to  reach  consensus  and  you  get 
more  insights  than  from  an  individual  interview  since  responses  spark  the  responses  of  others 
being  interviewed.  Responses  are  influenced  not  only  by  personal  thoughts  and  feelings,  but  also 
by  the  thoughts  and  feelings  of  other  participants. 

Don’t  Ask  Too  Many  Questions:  Your  interview  should  be  short.  Otherwise  participants  may 
lose  interest,  or  become  agitated  and  eventually  decline  to  continue.  Unlike  survey  items, 
interview  questions  are  phrased  to  elicit  an  extended  narrative.  Thus,  they  take  time  and  you  will 
not  have  time  to  ask  many  questions. 
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Decide  How  Rigorous  or  Flexible  Your  Questioning  Should  Be:  How  rigid  or  flexible  do  you 
want  your  questioning  to  be?  In  a  pretest  situation,  you  usually  keep  your  questioning  very 
flexible.  This  would  be  true  if  you  were  just  introduced  to  the  administrator  of  a  facility  at  which 
you  were  about  to  conduct  more  formal  interviews.  It  would  also  be  the  case  if  someone  from  a 
group  interview  wanted  to  speak  to  you  individually.  Sometimes  you  need  to  keep  your 
questions  nearly  identical  across  interviews,  especially  if  you  have  several  people  conducting 
interviews  and  consistency  is  desired.  In  such  a  case  you  would  memorize  them,  but  present 
them  in  a  conversational  manner  that  appears  unrehearsed  during  the  interview.  If  not,  it  may  be 
sufficient  to  develop  well  crafted  questions  beforehand  and  review  them  just  before  the 
interview,  writing  a  few  notes  to  remind  yourself  of  what  you  want  to  say.  Sometimes,  you  need 
just  enough  question  structure  to  get  those  being  interviewed  to  begin  talking,  and  then  you  base 
follow-up  questions  (probes)  almost  entirely  on  what  has  been  said.  At  other  times,  as  you  gain 
progressively  more  information  going  fi’om  one  interview  to  the  next,  your  questions,  or  how  you 
word  them,  may  change  due  to  your  improved  understanding  of  the  issues. 

Regardless  of  the  rigidity  or  flexibility  of  question  wording,  do  not  go  into  an  interview 
blind.  Even  anticipated  conversations  should  be  thought  through  beforehand.  Consider  ahead  of 
time  what  your  purpose  is,  what  you  intend  to  get  from  the  interview,  and  the  questions  you 
intend  to  ask  in  various  situations. 

Keep  Questions  Neutral:  How  questions  are  phrased  is  important.  Phrase  them  in  an  unbiased 
or  neutral  way:  one  that  does  not  predispose  a  certain  response  or  suggest  a  desired  answer. 

Use  Probes  to  Encourage  Elaboration:  Interview  questions  should  be  phrased  to  discourage 
brief  responses. 

Ensure  Understanding  Through  Clarification  and  Feedback:  Sometimes  participants  need 
clarification  about  what  you  said.  They  may  not  understand  the  meaning  of  a  question  as  you 
phrased  it  and  want  elaboration.  They  may  have  forgotten  what  you  asked.  They  may  be  unsure 
of  the  definition  of  a  term  you  used.  Prior  to  your  interview,  consider  how  to  clarify  or  elaborate 
upon  a  question,  or  define  a  term,  so  you  and  other  interviewers  can  do  so  effectively. 

Make  Note  Taking  and  Response  Coding  as  Simple  As  you  Can:  In  addition  to  writing  down 
interview  questions  and  having  them  available  during  the  interview,  do  what  you  can  to  make 
note  taking  and  data  gathering  simple,  fast,  and  effective.  In  many  situations  it  is  wise  to 
develop  a  response  form  that  allows  you  to  take  notes  or  annotate/code  responses  with  the  least 
amount  of  writing. 


Preparing  Yourself  to  Interview 

Know  Your  Questions  Well:  In  less  structured  interviews,  such  as  a  survey  pretest,  you  may 
not  care  if  your  questions  are  exactly  the  same  from  interview  to  interview.  You  could  review 
your  questions  beforehand  and  have  an  easy-to-follow  prompt  in  front  of  you  while  you 
interview.  This  prompt  could  be  part  of  a  simplified  interview  protocol.  Instead  of,  or  in 
addition  to  the  question  itself,  it  could  contain  a  bold  phrase  to  remind  you  of  the  question.  You 
should  know  your  questions  so  well  that  you  do  not  rely  much  on  your  notes.  Questions  should 
come  freely  and  sound  natural.  Do  not  be  a  slave  to  your  interview  protocol.  Retain  flexibility 
and  individuality.  Probes  can  help  you  introduce  flexibility  and  tailor  your  questions  to  specific 
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respondents.  Use  them  to  gain  greater  understanding.  You  can  generate  probable  probes  before 
going  into  the  interview  and  include  them  in  your  protocol.  Probes  usually  need  to  be  partly 
improvised  to  be  well  tailored  to  the  specific  comments  of  interviewees.  They  are  usually 
exploratory  requests  for  clarification  or  elaboration  of  the  prior  statements  and  have  to  be  crafted 
in  the  context  of  the  interview  itself. 

Practice  Your  Interview:  Practice  your  interview  after  refining  it  but  before  conducting 
interviews  in  the  field.  Have  a  colleague  assume  the  role  of  interviewee,  or  find  someone  from 
your  target  population  to  sit  through  the  interview.  Do  this  several  times.  Look  for 
shortcomings  in  your  interview  style  and  for  quirky  mannerisms  (we  all  have  them).  Do  you  ask 
questions  in  an  easy,  conversational  manner  without  stumbling  or  having  to  rely  too  much  on 
your  notes?  Are  you  friendly  rather  than  aloof?  Do  you  refrain  from  showing  anger  or  disdain? 
Do  you  avoid  leading  participants  to  respond  in  particular  ways?  Can  you  keep  yourself  from 
fiddling  with  your  hair,  appearing  nervous,  or  avoiding  eye  contact?  With  the  knowledge  gained 
from  the  videos  and  lots  of  practice,  you  can  probably  overcome  most  shortcomings. 

Schedule  Plenty  of  Time:  When  you  schedule  your  interviews,  you  will  specify  (or  negotiate) 
the  desired  number  of  people  per  interview  and  the  probable  length  of  each  interview  with  your 
POC.  In  a  group  interview,  the  size  should  be  about  5  to  10  or  12  people.  It  is  unwise  to 
interview  large  groups  at  once.  (In  a  pretest,  sometimes  you  have  to  anyway.)  I  recommend  that 
interviews  be  scheduled  for  between  thirty  minutes  to  one  hour,  with  some  leeway  to  continue 
longer  if  the  interview  is  going  well  and  the  participant  or  participants  are  agreeable. 

Regardless  of  whether  or  not  you  permit  the  interview  to  continue  longer  than  scheduled, 
schedule  more  time  than  you  need  and  allow  at  least  a  half  hour  between  interviews.  You  may 
not  have  complete  control  over  how  long  your  interview  takes.  For  instance,  the  person 
interviewed  may  be  interrupted,  and  you  may  have  to  “stop  the  clock”  for  a  while.  The  time 
between  interviews  can  be  valuable  to  you;  for  example,  to  write  notes  or  to  relax  since 
interviews  are  not  easy  and  can  be  exhausting.  If  you  use  interview  teams  as  recommended 
below  (see  Use  Two  Person  Teams),  it  is  advisable  for  the  team  members  to  discuss  interviews 
immediately  after  they  occur.  Interviewers  can  also  jot  down  or  elaborate  on  notes  between 
interviews  while  comments  are  fresh  in  their  minds.  The  bottom  line  is  to  schedule  more 
interview  time  than  you  expect  to  use  and  plenty  of  time  between  them.  This  not  only  helps  you; 
it  helps  the  interviewee  to  not  feel  rushed,  and  prevents  you  from  having  to  abruptly  terminate  an 
interview  because  the  next  person  or  group  is  waiting. 

Sometimes  interviewers  have  the  inconvenience  of  having  to  interview  at  several 
locations  in  an  unfamiliar  setting.  These  interviews  may  not  even  be  in  the  same  building  or  at 
the  same  site.  You  may  have  difficulty  going  from  one  interview  location  to  the  next  and  may 
even  get  lost  along  the  way.  In  such  a  situation,  scheduling  lots  of  time  between  interviews 
becomes  even  more  critical. 

Arrange  for  Comfortable  Surroundings:  If  possible,  conduct  the  interview  in  comfortable 
surroundings  and  away  from  the  person's  office.  You  do  not  want  the  interviewee  to  be 
distracted  by  phones  or  other  job  demands.  Keep  the  setting  as  intimate  as  possible.  Fit  the 
room  to  the  number  of  people  involved.  Do  not  interview  a  few  people  in  a  very  large  room. 
However,  the  room  should  not  be  so  small  that  people  feel  cramped  or  claustrophobic. 
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Use  Two  Person  Teams:  Do  not  gang  up  on  interview  participants.  For  instance,  in  most 
contexts,  4  or  5  people  should  not  interview  1  or  2  persons.  In  many  situations,  two-person 
teams  are  ideal.  The  two  people  usually  play  different  roles.  The  main  interviewer  is  the 
facilitator,  focusing  most  of  his  or  her  attention  on  the  person  being  interviewed  (active  listening, 
maintaining  eye  contact,  encouraging,  asking  questions  and  probes).  The  other  interviewer  may 
focus  on  note  taking  (see  cautionary  note  below)  while  still  being  attentive  to  the  person 
interviewed.  The  note  taker  is  not  restricted  from  asking  questions  (especially  probes),  but  this 
activity  is  secondary.  These  roles  can  be  switched  during  different  interviews. 

Sometimes  it  is  wise  not  to  take  notes  during  an  interview.  Note  taking  may  be 
distracting  to  the  person(s)  being  interviewed,  or  may  cause  them  to  restrict  what  they  say  for 
fear  of  disclosure.  Most  people  want  their  statements  not-for-attribution  and  note  taking 
suggests  that  their  statements  may  be  attributed  to  them.  This  is  a  judgment  call.  You  may  want 
to  stop  taking  notes  if  a  person  gets  into  something  very  personal  and  distressing  to  them,  or 
which  may  be  used  against  them. 

Present  A  Professional  Appearance:  Dress  professionally  for  the  interview  and  groom 
yourself  well.  Usually,  conservative  business  clothes  are  appropriate.  However,  if  those  being 
interviewed  might  feel  uncomfortable  around  interviewers  who  are  very  dressed  up,  dress  more 
casually.  You  want  to  lessen  status  differences  and  need  only  enough  status  to  establish  your 
credibility  as  an  interviewer.  Conform  to  the  dress  expectations  of  those  being  interviewed,  but 
dress  nicely. 


Conducting  the  Interview 


Motivate  Participation 

Greet  People  and  Introduce  Yourself;  As  people  come  to  the  interview,  greet  them  in  a  warm, 
friendly  manner.  Often,  it  is  wise  to  shake  hands  with  those  being  interviewed.  However,  if  the 
group  is  too  large,  or  you  think  they  may  be  uncomfortable  with  this  gesture,  do  not  do  it. 
Frequently,  people  are  already  seated  for  the  interview  and  it  may  be  cumbersome  to  shake 
everyone’s  hand.  You  can  at  least  wave  a  friendly  hello  as  you  are  introduced.  You  will  likely 
be  introduced  by  others  in  a  formal  way,  but  do  not  be  too  formal  yourself.  For  instance, 
introduce  yourself  by  your  first  name  instead  of  using  Mr.,  Ms.,  Dr.  or  Colonel. 

Knowing  the  names  of  the  persons  you  are  interviewing  is  advantageous,  but  it  may  not 
be  practical.  If  you  do  know  their  names,  use  this  knowledge  when  addressing  them.  However, 
beware— do  not  assume  you  can  call  everyone  by  their  first  names.  Some  people  would  consider 
this  an  affront;  especially  from  a  stranger  or  someone  they  consider  subordinate;  others  would 
welcome  it.  Use  good  judgment.  Perhaps  even  ask.  Especially  in  the  military,  use  “Sir”  or 
“Ma’am.” 

Indicate  the  Probable  Length  of  the  Interview:  The  person  or  persons  interviewed  may  be 
busy  and  believe  they  have  more  important  things  to  do  than  answer  your  questions.  Keep  the 
interview  short.  Be  honest  and  realistic  up-front  about  how  long  you  expect  it  to  take.  Thirty 
minutes  or  less  is  ideal,  but  an  hour  or  more  may  be  acceptable.  Participants  may  already  know 
the  expected  length  of  the  interview,  since  you  negotiated  it  when  you  scheduled  your 
interviews.  But  it  is  good  to  repeat  your  expectations  to  the  group.  If  they  believe  the  length  to 
be  reasonable,  this  will  heighten  their  motivation  to  participate. 
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Overcome  Reluctance  and  Heighten  Interest;  Establish  good  rapport  with  participants  early  in 
the  interview.  If  the  anyone  is  reluctant,  you  can  make  a  “conversion  attempt”  (jargon  for  trying 
to  change  the  person’s  minds  about  participating).  Briefly  give  them  an  idea  of  who  you  are. 
Explain  why  you  have  come  and  who  your  client  is.  Also  explain  what  the  interview  is  about, 
why  it  is  important,  and  why  their  participation  is  critical.  Your  first  few  minutes  are  critical-- 
you  must  convey  quickly  that  you  know  what  you  are  doing  and  capture  the  interest  of  those 
being  interviewed. 

Some  persons  interviewed  may  feel  ill  at  ease.  In  such  cases,  be  s5Tnpathetic  and 
friendly,  try  to  reduce  the  threat.  Put  the  persons  at  ease.  Make  it  clear  you  are  not  there  to 
judge  or  to  punish  but  to  gather  useful  information  in  a  way  that  will  help  rather  than  harm 
participants.  Sometimes  interviewees  may  be  more  shy  than  threatened  and  they  may  be 
unaccustomed  to  responding  to  questions  face-to-face.  Try  to  bring  such  people  out.  In 
establishing  rapport,  initially  ask  the  interviewee  a  few  easy  questions  about  themselves  or  their 
job  to  “break  the  ice,”  even  if  this  is  not  central  to  the  issues  you  want  to  cover. 

Sometimes  respondents  may  be  reluctant  to  participate  because  you  failed  to  convince 
them  of  the  importance  of  the  interview.  At  other  times  they  may  be  concerned  about  privacy  or 
confidentiality.  Probe  the  reasons  for  refusal  and  gently  do  what  you  can  to  heighten  their 
interest  and  allay  their  fears. 

One  way  to  get  people  interested  is  to  describe  how  their  input  should  help  bring  about 
constructive  change.  (Do  not  promise  this  unless  you  are  confident  such  change  will  occur.) 

You  can  also  offer  to  share  the  aggregate  results  of  the  interviews  and  explain  steps  your  client  is 
likely  to  take  to  resolve  the  issues  raised. 

Reward  Participation:  The  person  interviewed  should  feel  his  or  her  time  is  well  spent.  The 
rewards  of  participation  should  outweigh  the  costs.  Rewards  can  be  subtle:  anything  from  the 
pleasure  of  interacting  with  you  to  feeling  consulted  and  able  to  have  a  positive  influence. 

People  like  being  consulted;  it  suggests  they,  and  their  views,  are  valued.  You  reward 
participation  subtly  throughout  the  interview  by  listening  intently,  saying  things  like  “Uh  huh,” 
maintaining  eye  contact,  and  thanking  participants  for  sharing  their  views.  Remember,  however, 
that  you  must  remain  neutral  and  should  refrain  from  endorsing  what  is  being  said  (for  instance 
by  saying  “What  a  wonderful  idea”).  On  the  other  hand,  you  must  never  express  disdain  for 
what  is  being  said,  or  condemnation  of  the  interviewee’s  views  or  behaviors. 

Explain  Ethical  Safeguards: 

Voluntary  Participation:  Voluntary  participation  is  important.  Explain  early  in  the 
interview  that  participation  is  voluntary.  If  anyone  is  uncomfortable  and  expresses  a  desire  not 
to  participate,  gently  probe  why.  If  you  can  convince  them  to  stay,  fine;  if  not,  let  them  leave 
and  thank  them  for  coming. 

Confidentiality  and  Non-Attribution:  Emphasize  confidentiality  and  non-attribution 
early  in  your  introductoiy  comments.  If  anyone  is  concerned  about  confidentiality,  try  to  allay 
their  fears.  If  you  are  conducting  a  group  interview,  request  that  all  participants  agree  not  to 
attribute  specific  comments  to  individuals  when  they  leave  the  interview. 
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Be  Friendly  Yet  Professional:  As  you  transition  from  introductory  comments  to  interview 
questions,  continue  to  make  good  use  of  your  social  skills.  Be  friendly  and  warm  as  the 
interview  continues,  but  also  be  professional  and  purposeful.  Don't  be  chatty,  as  you  may  have 
been  before  the  interview  when  you  were  trying  to  build  rapport. 

Maintain  Subtle  Control:  As  the  interviewer,  you  are  “in  charge.”  You  have  the  right,  if  not 
the  obligation,  to  maintain  control  of  the  interview.  However,  you  should  be  subtle. 

Avoid  Rank  Intimidation:  In  most  organizational  settings,  hierarchy  and  status  are 
important.  Where  you  are  in  the  hierarchy  may  be  an  issue,  especially  in  the  military,  where 
“rank  has  its  privileges.”  Naturally,  all  persons  interviewed  should  be  treated  with  dignity  and 
respect.  However,  do  not  be  intimidated  by  rank.  Do  not  treat  those  of  higher  rank  with  special 
deference  in  the  interview.  The  same  courtesy  and  kindness  should  be  extended  to  all.  Consider 
all  views  equally  important  and  do  not  mentally  weight  comments  by  the  rank  of  the  persons 
making  them.  However,  rank  and  other  interviewee  characteristics  are  sometimes  noted  for 
demographic  comparisons.  People  of  different  grades  or  ranks--or  genders,  ages,  positions,  etc.— 
usually  have  different  views  and  it  is  informative  to  identify  these  differences. 

As  interviewer,  you  are  in  charge  of  the  interview.  However,  do  not  flaunt  your  power. 

If  you  are  of  higher  grade  or  rank  than  those  being  interviewed,  lessen  status  differences  by 
treating  others  as  your  equal  and  doing  other  things  to  de-emphasize  the  status  differential.  Do 
not  sit  at  the  head  of  the  table.  Encourage  others  to  call  you  by  your  first  name  and  downplay 
your  rank  or  title.  If  you  are  military,  consider  not  wearing  your  uniform.  On  the  other  hand, 
wearing  your  uniform  may  be  expected  and  may  help  you  establish  rapport  with  other  military 
members. 

Avoid  Authoritarian  Control:  You  are  just  a  visitor  when  conducting  interviews  at 
other  locations.  Your  sources  of  power  may  be  more  limited  than  in  your  own  organization.  Do 
not  alienate  those  being  interviewed  because  they  can  keep  their  mouths  shut  or  treat  you  like  a 
substitute  teacher.  Those  used  to  command  roles  may  need  to  be  more  subtle  than  usual  when 
trying  to  maintain  control  in  an  interview. 

Trying  to  be  in  charge  without  command  authority  can  be  difficult.  You  may  have 
power  based  on  the  prestige  of  your  organization  or  the  client  who  sent  you  to  interview.  You 
may  have  “expert  power”  because  you  are  reputed  to  know  what  you  are  doing.  You  may  have 
developed  a  reputation  as  a  caring  person  who  can  be  trusted  not  to  betray  confidences.  But  your 
main  source  of  power  is  the  liking,  acceptance  and  respect  you  generate  in  the  interview.  Use 
your  expertise,  charm,  communication  and  interpersonal  skills  to  maintain  control  without  being 
heavy  handed. 

Watch  for  Distortion:  Positive  Spin:  Watch  for  distortion.  Look  for  “walk-on- water” 
embellishments  of  accomplishments  or  capabilities.  People  may  make  things  sound  better  than 
they  are.  You  may  encounter  social  desirability  bias.  The  interviewee  may  want  to  please  you 
by  telling  you  what  they  think  you  want  to  hear  rather  than  accurately  stating  their  own  views. 
Alternatively,  an  interviewee  may  have  an  overly  negative  attitude,  finding  fault  with  everything. 
By  being  sensitive  to  these  possibilities,  and  by  interviewing  several  people  with  different  roles 
and  responsibilities,  you  can  get  a  more  balanced  picture. 


117 


Be  Sensitive  to  "Demand  Characteristics:"  Bias  or  Distraction:  face-to-face  contact 
provides  more  information  about  both  interviewer  and  interviewee.  Unfortunately,  the  presence 
of  an  interviewer  can  add  unwanted  "demand  characteristics"  to  the  situation.  Bias  can  be 
introduced.  As  discussed  earlier,  the  interviewee  may  want  to  respond  in  a  socially  desirable 
way.  The  interviewee  may  be  influenced  by  interviewer  characteristics  such  as:  gender;  race  or 
ethnicity;  social  class,  status,  organizational  position,  age,  attractiveness,  grooming,  or  manner  of 
dress.  Participants  may  be  offended  by  interviewers  who  are  abrasive,  insensitive,  haughty,  or 
rude. 

Be  Sensitive  to  Cultural  Differences 

Cultural  Diversity:  We  live  in  a  multicultural,  pluralistic  society  with  people  from 
different  ethnic,  racial,  national  and  spiritual  backgrounds.  They  are  of  different  genders  and 
ages  and  have  different  lifestyles,  political  views,  interests,  values  etc.  Interviewers  must  have 
knowledge  of  cultural  diversity  and  be  sensitive  to  multicultural  issues.  They  should  be  open 
and  flexible  enough  to  appreciate  diversity  and  considerate  enough  not  to  offend  representatives 
of  different  groups.  Interviewers  should  get  people  of  differing  backgrounds  or  lifestyles  to 
interact  with  them  and  with  each  other,  frankly  expressing  their  views. 

Language  Barriers:  The  interviewees  may  not  speak  English,  or  he  or  she  may  be  more 
comfortable  in  another  language.  If  this  the  case,  use  bilingual  interviewers  whenever  necessary. 

Common  Ground:  Establish  common  ground  between  those  being  interviewed  and 
those  conducting  the  interview.  A  pre-interview  chat  about  things  you  have  in  common  (sports, 
music,  a  family,  etc.)  may  help.  Usually  you  do  not  have  time  for  this,  or  you  may  have  little 
idea  of  what  you  have  in  common.  Outwardly  obvious  things  can  help  establish  common  ground 
(like  gender,  race,  age  or  military/civilian  status)  and  make  interviewees  more  comfortable.  If 
you  are  interviewing  a  diverse  workforce,  use  mixed  interviewer  teams  which  mirror  such 
diversity,  and  tailor  the  team  to  the  persons  being  interviewed. 

Maintain  Your  Neutrality:  Just  as  your  questions  must  be  neutral,  you  must  also  be  neutral. 
Your  personal  opinions  are  not  relevant.  Do  not  convey  your  own  views  or  you  may  bias 
responses.  Do  not  criticize  or  be  judgmental  about  the  views  or  behavior  of  interviewees. 
Likewise,  do  not  show  strong  endorsement.  Listen  attentively,  but  avoid  conveying  your  own 
enthusiasm  or  disdain  for  particular  responses. 

Handle  Abuse  by  Keeping  Your  Cool:  Do  not  be  surprised  if  you  are  verbally  attacked  during 
an  interview.  An  interviewee  may  object  to  something  you  say  and  lecture  you.  If  something 
like  this  happens  to  you,  patiently  listen  without  getting  angry  or  defensive.  Tiy  to  understand 
the  other  person’s  perspective  and  convey  that  understanding  without  being  argumentative. 

Within  limits,  take  the  verbal  abuse;  it  will  be  over  soon.  After  giving  the  person  reasonable 
time  to  express  his  or  her  concerns,  politely  but  assertively  return  to  the  interview  task.  If  a 
person’s  wrath  is  directed  to  the  interview  itself— for  instance,  it  is  considered  too  invasive,  or  a 
waste  of  time,  excuse  the  person.  Do  not  become  angry  or  unprofessional,  and  do  not  take  the 
hostility  jjersonally. 

Allow  For  Reflection  and  Probe  For  Clariflcation:  Interviewees  may  feel  pressured  to 
provide  a  quick  response  and,  having  little  time  for  deliberation,  may  answer  too  soon,  giving  a 
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short  or  superficial  response.  Allow  them  time  to  reflect,  and  probe  (ask  a  follow-up  question 
tailored  to  the  response)  if  the  answer  given  seems  insufficient. 

Listen  More  than  You  Speak:  Interviewers  must  be  articulate,  and  occasionally,  verbally 
assertive.  However,  it  is  not  the  interviewer’s  role  to  dominate  the  conversation.  Your  job  is  to 
get  others  to  speak  and  to  listen  for  the  meaning  in  what  they  say.  They  are  the  content  experts. 
Interviewers  should  talk  little  and  listen  a  lot.  They  should  also  avoid  interrupting  others  who 
are  speaking.  During  pauses,  the  interviewer  can  interject  a  comment,  probe  for  clarification,  or 
move  to  another  topic. 

Encourage  Everyone’s  Participation:  If  you  are  in  a  group  interview,  pay  attention  to  and 
encourage  everyone’s  participation.  Do  not  let  the  men  (or  women)  dominate,  or  the  highest- 
ranking,  or  the  managers,  or  the  loudest,  or  the  most  opinionated.  Encourage  shy  participants  to 
speak.  Ask  them  what  they  think  of  your  question  or  of  what  others  are  saying. 

Terminate  the  Interview:  At  the  beginning  of  the  interview  the  interviewer  should  indicate  the 
approximate  time  the  interview  will  take.  How  long  it  actually  takes  depends  on  how  well  it  is 
going  and  how  enthusiastically  engaged  participants  have  become.  If  you  exceed  the  allotted 
time,  ask  if  continuing  the  interview  is  acceptable,  unless  it  is  going  so  well  that  you  do  not  want 
to  break  the  flow.  If  you  must  interview  for  extended  periods,  schedule  multiple  sessions  over 
several  days.  Schedule  additional  interviews  in  advance  and  be  sensitive  to  the  interviewee's 
own  schedule.  Be  sensitive  also  to  subtle  cues  that  it  may  be  time  to  leave,  and  thank  people  for 
their  participation  when  you  terminate  the  interview.  Avoid  having  to  rush  people  out  so  the 
next  group  can  come  in. 


119 


Appendix  K:  Suggested  Questions  for  Pretest  Participants 

These  suggested  questions  would  be  asked  after  pretest  participants  have  taken  your 
survey.  Questioning  need  not  be  very  structured  and  you  need  not  ask  the  questions  in  exactly 
the  way  they  are  presented  here.  These  are  just  question  ideas.  Questions  like  these  could  be 
asked  of  individual  respondents,  respondents  in  larger  groups,  or  respondents  in  small,  informal, 
focus  groups.  Of  the  three  formats  the  focus  group  is  best  since  it  is  intimate,  relaxed,  and 
produces  a  synergistic  effect  by  encouraging  participants  to  reflect  and  comment  on  the  views  of 
other  respondents.  You  may  not  have  the  luxury  to  use  focus  groups.  For  example,  you  may 
only  have  participants  for  a  limited  time  and  need  to  interview  them  all  at  once  in  a  larger  group. 
Do  the  best  you  can. 

This  is  a  list  of  possible  questions  that  addresses  multiple  survey  and  interview  formats. 
Do  not  expect  to  ask  them  all.  Develop  additional  pretest  questions  of  special  interest  to  you. 

INTRODUCTORY  COMMENTS:  Explain  the  purpose  of  a  pretest  to  participants  and 
emphasize  how  you  need  their  help  to  improve  the  survey  before  it  goes  to  the  field.  Since  they 
represent  the  people  who  will  be  receiving  the  survey,  they  have  a  special  perspective  to  offer. 
Without  their  input,  the  survey  developer  could  not  be  sure  the  right  questions  are  being  asked 
and  if  they  communicate  well  to  respondents. 

LENGTH:  Have  participants  indicate  their  start/stop  times  unless  this  is  done  automatically  by 
your  computer  software  on  an  automated  survey.  Was  the  survey  too  long,  too  short,  or  about 
right?  About  how  long  did  it  take? 

OVERALL  REACTION  TO  SURVEY:  Generally,  what  did  participants  think  of  the  survey? 
Let  them  ventilate  here.  Do  not  become  annoyed  if  they  did  not  like  your  survey,  or  if  they 
express  concerns  about  confidentiality. 

OVERALL  REACTION  TO  TOPIC:  Generally,  what  did  they  think  about  the  content  or 
topic  of  the  survey?  Is  it  important  to  them? 

COVER  LETTER,  VOLUNTARY  PARTICIPATION  AND  CONFIDENTIALITY:  Was 
the  cover  letter  clear.  Did  it  convince  pretest  participants  that  the  survey  was  important?  Did 
they  understand  that  the  survey  was  voluntary  and  confidential?  What  do  these  terms  mean  to 
them?  Did  they  trust  that  individual  responses  would  be  kept  confidential?  How  important  is 
confidentiality  to  them? 

INSTRUCTIONS:  Were  the  instructions  easy  to  understand?  Did  they  understand  how  to  use 
the  answer  sheet  or  how  to  respond  via  computer?  Did  they  need  examples  of  the  right  and 
wrong  ways  to  mark  the  answer  sheet,  if  a  paper-and-pencil  format  was  used?  If  a  computer  was 
used,  was  there  too  much  information  on  each  computer  screen? 

ANSWER  SHEET :  Do  they  understand  what  they  should  not  do  with  the  answer  sheet  if  a 
paper-and-pencil  format  with  a  separate  sheet  was  used?  For  instance,  did  they  know  not  to  fold 
it,  staple  it,  place  stray  marks  on  it,  or  use  a  pen?  Was  the  sheet  easy  to  read? 

HELP  OR  HARM:  Do  they  believe  the  survey  will  be  used  to  help  them  and  their  coworkers 
(or  whomever)  or  to  hurt  them?  Do  they  think  that  management  (or  other  client)  will  take  their 


120 


responses  seriously  enough  to  make  needed  improvements?  Will  these  changes  positively  affect 
them? 

PARTICIPATION  WORTH  THE  TIME:  Was  the  survey  worth  taking— worth  their  time  to 
complete? 

RELUCTANCE  TO  PARTICIPATE:  Were  they  reluctant  to  take  the  survey,  or 
offended/annoyed  by  any  of  its  content?  Were  items  too  intrusive?  Were  there  any  items  they 
felt  they  should  not  answer?  Why?  What  items,  if  any,  did  they  skip?  Were  they  consciously 
deceptive  in  responding  to  any  items?  Which  ones? 

CHANGES,  ADDITIONS,  DELETIONS:  Were  there  any  items  (or  words  or  phrases)  they  did 
not  understand?  Would  they  recommend  wording  any  items  differently,  either  to  improve  their 
clarity  or  to  lessen  their  offensiveness?  What  items  or  other  content  would  they  add  to  the 
survey  or  delete  from  it?  Why? 

APPROPRIATENESS  OF  RESPONSE  OPTIONS:  Were  the  response  options  complete  and 
appropriate?  (Avoid  jargon  such  as  exhaustive  and  mutually  exclusive.)  Did  they  understand 
how  to  use  common  response  scales?  Did  they  like  the  scales  used?  Did  they  find  themselves 
not  having  an  opinion  or  not  knowing  how  to  respond?  Were  the  number  of  common  scales  used 
too  many  or  just  about  right.  Were  there  other  responses  they  may  have  wanted  to  provide  to 
any  item?  Which  ones?  What  are  these  alternative  responses?  Did  they  like  having  negative 
items  included  on  the  survey?  Why? 

COMMENTS  PAGE;  Did  they  want  to  write  on  the  comments  page?  What  would  they  say? 
Was  there  sufficient  room  for  comments,  either  in  response  to  specific  items,  or  at  the  end  of  the 
survey? 

FLAWLESS  COMPUTER  OPERATION:  If  the  survey  was  computerized,  was  the  operation 
of  the  survey  flawless.  Did  it  branch  in  a  way  that  was  transparent  to  the  user?  That  is,  did  it 
turn  out  that  members  of  the  appropriate  subgroup  got  the  items  intended  for  them  and  others  did 
not? 

FORMAT  PREFERENCE:  If  the  survey  is  automated,  did  participants  prefer  the  automated 
format  to  paper-and-pencil  surveys  they  had  taken  in  the  past?  If  the  survey  used  the  Computer- 
Assisted  Telephone  Interview  (CATI)  approach,  was  it  preferred  to  other  formats? 

INTERVIEWER  SKILLS:  If  a  CATI  survey  or  face-to-face  interview  was  used,  was  the 
interviewer  particularly  skilled?  Skill  dimensions  could  include,  but  not  be  limited  to: 

•  success  with  a  conversion  attempt  (i.e.,  convincing  reluctant  people  to  participate), 

•  friendliness,  warmth,  and  empathy, 

•  flexibility, 

•  knowledge  of  the  topic, 

•  questioning  skill  (including  probes), 

•  clarification  skill, 

•  ability  to  listen, 

•  remaining  calm,  unflustered  and  non-judgmental,  and 

•  knowing  when  and  how  to  terminate  the  interview. 
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Was  the  interviewer  obnoxious,  pushy  or  otherwise  annoying?  What  advice  would  you  give  the 
interviewer  to  improve  his  or  her  future  performance? 

OTHER  CONCERNS  OR  COMMENTS:  Are  their  other  concems/comments  about  the  topic 
or  about  the  survey  they  would  like  to  make? 

THANK  YOU:  Thank  pretest  participants  for  their  assistance. 
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